Oct 31 15:59:41.353332 ip-10-0-129-180 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 15:59:41.353343 ip-10-0-129-180 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 15:59:41.353350 ip-10-0-129-180 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 15:59:41.353666 ip-10-0-129-180 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 15:59:51.353435 ip-10-0-129-180 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 15:59:51.353452 ip-10-0-129-180 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 624b77bdeb2c4453badb6239a90b2113 --
Oct 31 16:01:53.125839 ip-10-0-129-180 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 16:01:53.500377 ip-10-0-129-180 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:01:53.500377 ip-10-0-129-180 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 16:01:53.500377 ip-10-0-129-180 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:01:53.500377 ip-10-0-129-180 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:01:53.506479 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506400    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.506479 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506479    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.506479 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506483    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506487    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506490    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506494    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506499    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506502    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506505    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506508    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506511    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506514    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506517    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506525    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506528    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506531    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506533    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506536    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506539    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506541    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506544    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.506592 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506548    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506552    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506555    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506573    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506577    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506581    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506585    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506588    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506590    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506593    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506596    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506599    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506603    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506606    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506608    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506611    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506615    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506618    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506620    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.507038 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506623    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506625    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506628    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506630    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506633    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506635    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506638    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506640    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506643    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506646    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506649    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506651    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506654    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506656    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506660    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506664    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506666    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506669    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506672    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506675    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.507486 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506677    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506680    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506683    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506685    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506688    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506691    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506693    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506696    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506698    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506701    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506705    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506707    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506710    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506712    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506715    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506718    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506721    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506724    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506726    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.507970 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506728    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506731    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506733    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506736    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506738    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506741    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506743    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506746    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506750    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506752    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506755    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506758    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506760    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506763    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506765    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506768    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506770    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506773    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506776    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506779    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.508412 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506782    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506784    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506787    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506789    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506794    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506796    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.506799    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507556    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507577    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507582    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507586    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507589    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507592    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507594    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507597    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507600    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507602    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507605    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507608    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507611    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.508919 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507614    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507617    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507620    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507623    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507627    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507631    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507636    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507640    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507644    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507647    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507652    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507655    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507658    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507660    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507663    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507666    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507669    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507672    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507675    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.509381 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507678    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507681    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507683    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507686    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507688    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507691    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507693    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507696    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507698    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507702    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507704    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507707    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507710    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507713    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507716    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507719    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507723    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507725    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507728    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.509866 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507746    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507751    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507755    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507759    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507763    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507766    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507769    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507771    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507774    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507776    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507779    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507782    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507785    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507788    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507791    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507793    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507796    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507798    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507801    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507803    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.510450 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507806    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507809    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507811    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507814    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507817    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507819    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507824    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507828    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507832    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507835    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507839    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507843    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507848    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507852    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507855    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507859    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507863    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507867    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507871    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507874    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.511298 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507878    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507882    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507886    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507889    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507893    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507897    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507901    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507905    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507910    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507914    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507918    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507922    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507926    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507930    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.507934    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508446    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508452    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508455    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508458    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508461    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.511865 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508463    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508466    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508469    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508472    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508475    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508478    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508480    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508483    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508485    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508488    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508490    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508493    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508496    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508498    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508500    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508503    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508506    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508509    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508512    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508515    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.512342 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508517    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508520    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508522    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508525    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508527    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508530    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508532    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508535    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508538    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508541    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508543    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508546    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508548    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508551    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508553    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508556    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508575    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508579    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508581    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508584    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.512829 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508587    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508589    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508592    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508595    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508597    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508600    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508604    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508608    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508611    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508614    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508618    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508621    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508623    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508626    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508628    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508631    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508634    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508637    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.513302 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508639    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508642    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508645    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508648    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508650    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508653    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508655    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508658    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508661    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508663    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508666    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508668    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508671    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508674    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508676    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508679    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508681    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508683    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508686    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508689    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.513754 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508691    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508694    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508697    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508699    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508702    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508705    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508710    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508714    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508717    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508722    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508725    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508727    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508730    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508732    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508735    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508738    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508741    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508744    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508746    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.514267 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508749    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508752    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508754    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.508758    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509619    2587 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509629    2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509636    2587 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509640    2587 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509645    2587 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509648    2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509656    2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509661    2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509664    2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509667    2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509670    2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509674    2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509677    2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509680    2587 flags.go:64] FLAG: --cgroup-root=""
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509683    2587 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509690    2587 flags.go:64] FLAG: --client-ca-file=""
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509693    2587 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509697    2587 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509703    2587 flags.go:64] FLAG: --cluster-domain=""
Oct 31 16:01:53.514736 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509706    2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509709    2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509712    2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509716    2587 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509720    2587 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509723    2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509727    2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509730    2587 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509733    2587 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509736    2587 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509739    2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509742    2587 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509745    2587 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509750    2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509753    2587 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509756    2587 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509759    2587 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509762    2587 flags.go:64] FLAG: --enable-server="true"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509765    2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509769    2587 flags.go:64] FLAG: --event-burst="100"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509772    2587 flags.go:64] FLAG: --event-qps="50"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509775    2587 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509779    2587 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509781    2587 flags.go:64] FLAG: --eviction-hard=""
Oct 31 16:01:53.515304 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509786    2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509789    2587 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509792    2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509795    2587 flags.go:64] FLAG: --eviction-soft=""
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509800    2587 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509803    2587 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509807    2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509811    2587 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509815    2587 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509818    2587 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509820    2587 flags.go:64] FLAG: --feature-gates=""
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509825    2587 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509828    2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509832    2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509835    2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509838    2587 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509841    2587 flags.go:64] FLAG: --help="false"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509844    2587 flags.go:64] FLAG: --hostname-override="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509848    2587 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509851    2587 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509854    2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509857    2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509861    2587 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 16:01:53.515911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509864    2587 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509867    2587 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509870    2587 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509873    2587 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509877    2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509880    2587 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509883    2587 flags.go:64] FLAG: --kube-reserved=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509886    2587 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509889    2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509893    2587 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509896    2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509898    2587 flags.go:64] FLAG: --lock-file=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509902    2587 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509906    2587 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509909    2587 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509915    2587 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509919    2587 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509922    2587 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509925    2587 flags.go:64] FLAG: --logging-format="text"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509928    2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509931    2587 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509934    2587 flags.go:64] FLAG: --manifest-url=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509937    2587 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509943    2587 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509946    2587 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 16:01:53.516461 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509950    2587 flags.go:64] FLAG: --max-pods="110"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509953    2587 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509956    2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509959    2587 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509962    2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509965    2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509968    2587 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509972    2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509979    2587 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509982    2587 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509985    2587 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509988    2587 flags.go:64] FLAG: --pod-cidr=""
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509991    2587 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509994    2587 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.509997    2587 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510000    2587 flags.go:64] FLAG: --port="10250"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510002    2587 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510005    2587 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0cc520d52fbe906ec"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510008    2587 flags.go:64] FLAG: --qos-reserved=""
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510012    2587 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510016    2587 flags.go:64] FLAG: --register-node="true"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510019    2587 flags.go:64] FLAG: --register-with-taints=""
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510023    2587 flags.go:64] FLAG: --registry-burst="10"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510059    2587 flags.go:64] FLAG: --registry-qps="5"
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510218    2587 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510245    2587 flags.go:64] FLAG: --reserved-memory=""
Oct 31 16:01:53.517062 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510256    2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510263    2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510269    2587 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510276    2587 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510282    2587 flags.go:64] FLAG: --runonce="false"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510288    2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510294    2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510308    2587 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510314    2587 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510319    2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510325    2587 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510331    2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510336    2587 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510342    2587 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510348    2587 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510358    2587 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510363    2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510368    2587 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510373    2587 flags.go:64] FLAG: --system-cgroups=""
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510378    2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510402    2587 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510408    2587 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510419    2587 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510428    2587 flags.go:64] FLAG: --tls-min-version=""
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510433    2587 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 16:01:53.517827 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510438    2587 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510443    2587 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510448    2587 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510453    2587 flags.go:64] FLAG: --v="2"
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510461    2587 flags.go:64] FLAG: --version="false"
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510473    2587 flags.go:64] FLAG: --vmodule=""
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510485    2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.510492    2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.510976    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.510984    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.510990    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511011    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511027    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511032    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511081    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511251    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511258    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511261    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511264    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511267    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511270    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.518418 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511272    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511276    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511279    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511282    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511285    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511290    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511295    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511299    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511302    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511305    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511309    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511313    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511319    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511323    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511326    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511329    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511331    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511334    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511337    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.519017 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511340    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511343    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511346    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511350    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511353    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511356    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511359    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511362    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511365    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511367    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511370    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511373    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511376    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511378    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511381    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511384    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511388    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511395    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511401    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.519461 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511405    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511408    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511410    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511413    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511416    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511419    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511421    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511425    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511427    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511430    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511433    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511437    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511440    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511444    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511447    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511449    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511452    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511455    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511458    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511460    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.519935 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511464    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511468    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511472    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511476    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511480    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511484    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511489    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511491    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511494    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511497    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511500    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511502    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511505    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511508    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511511    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511513    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511516    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511518    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511521    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511524    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.520416 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511526    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511529    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511532    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511534    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511537    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511540    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511543    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511546    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511549    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511552    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511557    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511573    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511578    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511582    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511585    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.520894 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.511623    2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511667    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511673    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511677    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511680    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511682    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511685    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511688    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511690    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511693    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511695    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511698    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511701    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511703    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511706    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511708    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511711    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511713    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511716    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.521251 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511718    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511722    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511727    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511732    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511737    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511739    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511742    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511745    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511747    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511750    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511753    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511755    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511758    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511761    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511764    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511766    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511770    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511773    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511776    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.521719 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511779    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511781    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511784    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511786    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511789    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511791    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511794    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511796    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511799    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511804    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511808    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511813    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511817    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511820    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511822    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511825    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511830    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511833    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511835    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511838    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.522162 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511840    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511843    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511846    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511849    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511851    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511854    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511858    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511860    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511864    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511868    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511871    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511874    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511877    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511882    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511886    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511892    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511896    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511899    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511902    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.522633 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511904    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511907    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511909    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511912    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511914    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511917    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511919    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511922    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511924    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511928    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511931    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511934    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511936    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511939    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511941    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511946    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511948    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511951    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511953    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.523117 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511956    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511960    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511964    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511969    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511973    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511976    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511978    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511981    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511983    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511986    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.511990    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.519041    2587 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.519054    2587 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.519821    2587 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.519827    2587 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519894    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519899    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519902    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519905    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519908    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.523618 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519911    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519915    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519920    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519923    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519926    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519928    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519931    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519934    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519938    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519940    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519943    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519945    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519948    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519951    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519954    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519957    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519959    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519962    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519965    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.524086 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519967    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519970    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519973    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519976    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519979    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519981    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519984    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519987    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519989    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519992    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519995    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519997    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.519999    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520002    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520004    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520007    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520009    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520012    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520014    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520017    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.524545 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520019    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520022    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520025    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520027    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520030    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520033    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520035    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520038    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520040    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520043    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520045    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520048    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520050    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520053    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520055    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520058    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520066    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520070    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520072    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520074    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.525040 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520077    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520080    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520082    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520085    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520087    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520090    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520092    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520095    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520097    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520100    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520102    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520105    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520107    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520110    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520113    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520115    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520118    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520120    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520123    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520125    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.525503 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520128    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520130    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520133    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520136    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520138    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520141    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520143    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520146    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520148    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520152    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520154    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520157    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520159    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520162    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520164    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520167    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520169    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520172    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520176    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.526008 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520180    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520183    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520185    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.520213    2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520366    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520371    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520374    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520377    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520380    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520382    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520385    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520388    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520390    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520393    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520395    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520398    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.526463 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520400    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520402    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520405    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520407    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520410    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520414    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520418    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520422    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520425    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520428    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520432    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520435    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520438    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520441    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520443    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520446    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520448    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520451    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520453    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.526899 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520456    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520459    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520461    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520464    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520467    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520469    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520472    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520474    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520477    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520479    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520482    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520484    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520487    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520489    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520492    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520494    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520497    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520499    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520503    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.527367 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520507    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520511    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520515    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520519    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520524    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520528    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520532    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520537    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520539    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520542    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520545    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520547    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520551    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520553    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520556    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520570    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520573    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520575    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520578    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.527831 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520580    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520583    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520585    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520588    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520590    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520593    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520596    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520598    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520601    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520604    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520606    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520609    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520611    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520614    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520616    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520619    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520622    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520625    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520628    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520630    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.528352 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520635    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520638    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520641    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520643    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520646    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520648    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520651    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520653    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520656    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520658    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520661    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520663    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520666    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520668    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520671    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520673    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.528826 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520676    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.520699    2587 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520732    2587 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520735    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520738    2587 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520741    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520744    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520747    2587 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520749    2587 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520752    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520754    2587 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520757    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520759    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520762    2587 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520765    2587 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520768    2587 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:53.529195 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520774    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520776    2587 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520779    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520781    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520784    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520786    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520789    2587 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520791    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520794    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520796    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520799    2587 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520801    2587 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520804    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520806    2587 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520809    2587 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520812    2587 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520814    2587 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520817    2587 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520819    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:53.529602 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520822    2587 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520824    2587 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520827    2587 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520829    2587 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520832    2587 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520835    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520837    2587 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520839    2587 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520842    2587 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520844    2587 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520847    2587 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520850    2587 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520853    2587 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520856    2587 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520860    2587 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520865    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520868    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520871    2587 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520873    2587 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:53.530043 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520876    2587 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520878    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520881    2587 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520883    2587 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520886    2587 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520888    2587 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520891    2587 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520894    2587 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520897    2587 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520900    2587 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520902    2587 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520905    2587 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520908    2587 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520910    2587 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520913    2587 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520915    2587 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520920    2587 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520922    2587 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520925    2587 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520928    2587 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:53.530550 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520930    2587 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520933    2587 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520936    2587 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520938    2587 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520941    2587 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520945    2587 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520948    2587 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520952    2587 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520955    2587 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520958    2587 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520961    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520964    2587 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520966    2587 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520969    2587 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520971    2587 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520974    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520976    2587 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520979    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520982    2587 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:53.531084 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520984    2587 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520987    2587 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520990    2587 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520992    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520995    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.520997    2587 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521000    2587 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521003    2587 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521005    2587 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521009    2587 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521012    2587 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521014    2587 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521017    2587 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521020    2587 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:53.521023    2587 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.521070    2587 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.523197    2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.524040    2587 server.go:1014] "Starting client certificate rotation"
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.524137    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 16:01:53.531521 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.524164    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 16:01:53.531985 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.526049    2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:01:53.531985 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.530164    2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:01:53.538314 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.538293    2587 log.go:25] "Validated CRI v1 runtime API"
Oct 31 16:01:53.542269 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.542253    2587 log.go:25] "Validated CRI v1 image API"
Oct 31 16:01:53.543047 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.543022    2587 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 16:01:53.547661 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.547641    2587 fs.go:135] Filesystem UUIDs: map[113b7fbb-6d32-47d9-bbb4-9a3398174aab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 92ef52f4-ec37-4675-843c-672a3f74d545:/dev/nvme0n1p3]
Oct 31 16:01:53.547720 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.547659    2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 16:01:53.552697 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.552594    2587 manager.go:217] Machine: {Timestamp:2026-10-31 16:01:53.551300464 +0000 UTC m=+0.313550535 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082602 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec222fdb132d150bfe59a819f8d49ade SystemUUID:ec222fdb-132d-150b-fe59-a819f8d49ade BootID:624b77bd-eb2c-4453-badb-6239a90b2113 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ce:cf:db:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ce:cf:db:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:b5:86:fb:5b:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 16:01:53.553255 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.553246    2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 16:01:53.553334 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.553323    2587 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 16:01:53.556528 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.556503    2587 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 16:01:53.556703 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.556528    2587 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-180.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 16:01:53.556797 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.556707    2587 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 16:01:53.556797 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.556719    2587 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 16:01:53.556797 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.556771    2587 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:01:53.556897 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.556797    2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:01:53.558003 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.557989    2587 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 16:01:53.562759 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.562744    2587 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 16:01:53.562893 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.562882    2587 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 16:01:53.563672 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.563661    2587 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 16:01:53.563726 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.563678    2587 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 16:01:53.563726 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.563694    2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 16:01:53.563726 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.563709    2587 kubelet.go:405] "Adding apiserver pod source"
Oct 31 16:01:53.563726 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.563720    2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 16:01:53.564094 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.564062    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:01:53.566141 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.566117    2587 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 16:01:53.572306 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.572294    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:01:53.573191 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573178    2587 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 16:01:53.573229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573211    2587 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 16:01:53.573361 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573350    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 16:01:53.573390 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573367    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 16:01:53.573390 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573373    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 16:01:53.573390 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573379    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 16:01:53.573390 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573384    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 16:01:53.573390 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573389    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 16:01:53.573958 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573949    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 16:01:53.573988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573960    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 16:01:53.573988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573967    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 16:01:53.573988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573973    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 16:01:53.573988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573981    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 16:01:53.574108 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.573991    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 16:01:53.574624 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.574614    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 16:01:53.574655 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.574626    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 16:01:53.578282 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.578267    2587 server.go:1263] "Started kubelet"
Oct 31 16:01:53.578377 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.578355    2587 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 16:01:53.578422 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.578367    2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 16:01:53.578451 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.578438    2587 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 16:01:53.579134 ip-10-0-129-180 systemd[1]: Started Kubernetes Kubelet.
Oct 31 16:01:53.580005 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.579984    2587 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 16:01:53.581788 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.581771    2587 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 16:01:53.588321 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.588301    2587 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 16:01:53.589612 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.589591    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 16:01:53.590187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.590171    2587 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 16:01:53.590287 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.590267    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.590861 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.590840    2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 16:01:53.590861 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.590841    2587 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 16:01:53.590984 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.590873    2587 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 16:01:53.590984 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.590970    2587 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 16:01:53.590984 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.590981    2587 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 16:01:53.591076 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591049    2587 factory.go:55] Registering systemd factory
Oct 31 16:01:53.591076 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591067    2587 factory.go:223] Registration of the systemd container factory successfully
Oct 31 16:01:53.591280 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591266    2587 factory.go:153] Registering CRI-O factory
Oct 31 16:01:53.591331 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591283    2587 factory.go:223] Registration of the crio container factory successfully
Oct 31 16:01:53.591389 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591378    2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 16:01:53.591437 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591399    2587 factory.go:103] Registering Raw factory
Oct 31 16:01:53.591437 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.591413    2587 manager.go:1199] Started watching for new ooms in manager
Oct 31 16:01:53.592389 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.592375    2587 manager.go:319] Starting recovery of all containers
Oct 31 16:01:53.601420 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.601403    2587 manager.go:324] Recovery completed
Oct 31 16:01:53.606995 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.606982    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.608943 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.608839    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.608943 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.608877    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.608943 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.608891    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.609460 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.609444    2587 cpu_manager.go:226] "Starting" policy="none"
Oct 31 16:01:53.609460 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.609453    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.609460 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.609457    2587 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 16:01:53.609642 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.609479    2587 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 16:01:53.610139 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.610126    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.610185 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.610151    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.610185 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.610163    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.611822 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.611809    2587 policy_none.go:50] "Start"
Oct 31 16:01:53.611867 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.611825    2587 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 16:01:53.611867 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.611835    2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 16:01:53.613851 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.613840    2587 policy_none.go:44] "Start"
Oct 31 16:01:53.690549 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.690521    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.715354 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.715312    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 16:01:53.716466 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.716450    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 16:01:53.716585 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.716470    2587 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 16:01:53.716585 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.716490    2587 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 16:01:53.716585 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.716528    2587 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 16:01:53.721882 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.720985    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-180.us-west-2.compute.internal.18e3a8a767cf5723  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-180.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-180.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:53.578235683 +0000 UTC m=+0.340485754,LastTimestamp:2026-10-31 16:01:53.578235683 +0000 UTC m=+0.340485754,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-180.us-west-2.compute.internal,}"
Oct 31 16:01:53.754529 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.754451    2587 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 16:01:53.760201 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760185    2587 manager.go:341] "Starting Device Plugin manager"
Oct 31 16:01:53.760298 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.760220    2587 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 16:01:53.760298 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760229    2587 server.go:87] "Starting device plugin registration server"
Oct 31 16:01:53.760493 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760469    2587 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 16:01:53.760614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760483    2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 16:01:53.760614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760574    2587 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 16:01:53.760703 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760650    2587 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 16:01:53.760703 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.760659    2587 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 16:01:53.761181 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.761161    2587 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 16:01:53.761265 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.761208    2587 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.769823 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.769779    2587 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-180.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 16:01:53.769823 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.769797    2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-180.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 16:01:53.817509 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.817478    2587 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"]
Oct 31 16:01:53.817645 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.817543    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.818448 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.818434    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.818505 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.818463    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.818505 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.818473    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.820792 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.820780    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.820950 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.820935    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.821020 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.820974    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.821447 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.821430    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.821523 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.821460    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.821523 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.821475    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.821523 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.821512    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.821649 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.821532    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.821649 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.821542    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.823714 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.823653    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.823802 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.823733    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.824398 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.824377    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.824398 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.824398    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.824513 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.824407    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.849847 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.849819    2587 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 16:01:53.849949 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.849889    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c8l8k"
Oct 31 16:01:53.855296 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.855279    2587 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.857840 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.857822    2587 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.860874 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.860862    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.861607 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.861594    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.861658 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.861626    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.861658 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.861641    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.861719 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.861661    2587 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.864616 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.864598    2587 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-180.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 16:01:53.864678 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.864599    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-180.us-west-2.compute.internal.18e3a8a769a2952c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-180.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-180.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-180.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:53.608856876 +0000 UTC m=+0.371106951,LastTimestamp:2026-10-31 16:01:53.608856876 +0000 UTC m=+0.371106951,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-180.us-west-2.compute.internal,}"
Oct 31 16:01:53.864739 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.864721    2587 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 16:01:53.864782 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.864758    2587 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 16:01:53.893160 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.893138    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.893226 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.893163    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29bfc3859f40051644e6484149c81648-config\") pod \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"29bfc3859f40051644e6484149c81648\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.893226 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.893180    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.931018 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.930997    2587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:53.937131 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.937115    2587 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.937172 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:53.937138    2587 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-180.us-west-2.compute.internal\": node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.993630 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.993606    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.993758 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.993632    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.993758 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.993651    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29bfc3859f40051644e6484149c81648-config\") pod \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"29bfc3859f40051644e6484149c81648\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.993758 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.993722    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.993758 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.993734    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:53.993921 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:53.993791    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29bfc3859f40051644e6484149c81648-config\") pod \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"29bfc3859f40051644e6484149c81648\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:54.051379 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:54.051354    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:54.152220 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:54.152192    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:54.159503 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:54.159480    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:54.161630 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:54.161611    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:54.253407 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:54.253245    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:54.354217 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:54.354140    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.869818 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:55.869794    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.938654 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:55.938625    2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 16:01:55.939103 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:55.938754    2587 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 16:01:55.939103 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:55.938829    2587 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7918a53732334b8eb1150a335f38936-a35c44805d999c48.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-180.us-west-2.compute.internal.18e3a8a769a2952c\": read tcp 10.0.129.180:46312->3.148.49.148:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-180.us-west-2.compute.internal.18e3a8a769a2952c  default   5072 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-180.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-180.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-180.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:53 +0000 UTC,LastTimestamp:2026-10-31 16:01:53.82144732 +0000 UTC m=+0.583697397,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-180.us-west-2.compute.internal,}"
Oct 31 16:01:55.969890 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:55.969845    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.070675 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.070600    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.171708 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.171681    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.269987 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.269928    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 15:56:55 +0000 UTC" deadline="2028-06-14 02:09:58.292985394 +0000 UTC"
Oct 31 16:01:56.269987 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.269976    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14194h8m2.023012837s"
Oct 31 16:01:56.270182 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.270008    2587 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:56.272142 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.272126    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.330027 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.329962    2587 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:56.330838 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.330821    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:56.330937 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.330850    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:56.330937 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.330861    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:56.372289 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.372267    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.472888 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.472855    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.573526 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.573497    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.674112 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:56.674078    2587 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:01:56.704765 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:56.704727    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bfc3859f40051644e6484149c81648.slice/crio-219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4 WatchSource:0}: Error finding container 219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4: Status 404 returned error can't find the container with id 219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4
Oct 31 16:01:56.704992 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:56.704979    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bd4376b5e50178846f4b8cc0416cb3.slice/crio-3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d WatchSource:0}: Error finding container 3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d: Status 404 returned error can't find the container with id 3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d
Oct 31 16:01:56.709246 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.709230    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:01:56.753448 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.753419    2587 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:56.805568 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.805542    2587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:56.865580 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.865518    2587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:01:56.867295 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.867279    2587 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:01:56.966241 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.966212    2587 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:56.966241 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.966212    2587 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:56.966664 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.966216    2587 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:01:56.979364 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:56.979346    2587 apiserver.go:52] "Watching apiserver"
Oct 31 16:01:57.005539 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.005512    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 16:01:57.081836 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.081810    2587 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 16:01:57.082137 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.082114    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5n9tx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-cfrr7","kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk","openshift-cluster-node-tuning-operator/tuned-vv5k4","openshift-dns/node-resolver-c6ssr","openshift-image-registry/node-ca-cg6j2","openshift-ovn-kubernetes/ovnkube-node-j6bw8","kube-system/konnectivity-agent-b4fr7","openshift-multus/multus-additional-cni-plugins-qhr6p","openshift-multus/network-metrics-daemon-h6rw9","openshift-multus/multus-gk2gp","openshift-network-operator/iptables-alerter-2fr6l"]
Oct 31 16:01:57.085801 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.085781    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.087762 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.087743    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.089838 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.089819    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:01:57.089918 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.089890    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:01:57.091898 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.091878    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.094033 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.094019    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.096094 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.096078    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.098398 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.098382    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.100701 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.100686    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.102944 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.102931    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.105463 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.105437    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.105534 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.105510    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:01:57.107726 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.107712    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.109803 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.109777    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.109887 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.109837    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:01:57.117896 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.117873    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.117988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.117902    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88955cf1-7c74-4198-a5a6-762fc37473f9-agent-certs\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.117988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.117919    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cnibin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.117988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.117933    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.117988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.117955    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhms\" (UniqueName: \"kubernetes.io/projected/251fef43-4a22-4508-9e24-166a266d95a0-kube-api-access-svhms\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118012    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-host-slash\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118061    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-dbus\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118099    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-conf\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118125    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-lib-modules\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118160    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-script-lib\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118186    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.118229 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118210    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-kubelet\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118229    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118254    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-device-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118273    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-slash\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118286    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd2188c-94c4-4700-b0ff-9b7cf279911c-host\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118309    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-tmp\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118367    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-var-lib-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118394    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118410    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118429    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da23b925-afa4-4246-a9f9-38b52246d95a-hosts-file\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118524    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cni-binary-copy\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.118552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118546    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-etc-kubernetes\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118568    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-registration-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118591    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-sys-fs\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118608    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-kubernetes\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118627    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118647    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4x8\" (UniqueName: \"kubernetes.io/projected/bdd2188c-94c4-4700-b0ff-9b7cf279911c-kube-api-access-dg4x8\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118681    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-sys\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118705    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-etc-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118749    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-multus\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118764    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-hostroot\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118778    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-multus-certs\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118790    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118803    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysconfig\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118817    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-tuned\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118835    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-systemd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118851    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119056 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118870    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd2188c-94c4-4700-b0ff-9b7cf279911c-serviceca\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118883    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-modprobe-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118903    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-run\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118934    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-netd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118952    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.118984    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzk2\" (UniqueName: \"kubernetes.io/projected/68e7747c-9ba3-420e-aba4-25f0308c6a1e-kube-api-access-9xzk2\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119006    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da23b925-afa4-4246-a9f9-38b52246d95a-tmp-dir\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119020    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-var-lib-kubelet\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119038    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-node-log\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119058    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-iptables-alerter-script\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119077    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-socket-dir-parent\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119093    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-host\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119108    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-netns\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119122    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-ovn\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119135    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dcn\" (UniqueName: \"kubernetes.io/projected/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-kube-api-access-x5dcn\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119148    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghqcf\" (UniqueName: \"kubernetes.io/projected/da23b925-afa4-4246-a9f9-38b52246d95a-kube-api-access-ghqcf\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.119820 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119160    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-systemd\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119175    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119195    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-systemd-units\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119218    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119234    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-system-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119261    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-netns\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119277    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-socket-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119296    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119310    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/b2beb745-0c6b-4060-86fd-0e31575d1342-kube-api-access-vzvqg\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119332    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-daemon-config\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119349    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-env-overrides\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119362    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88955cf1-7c74-4198-a5a6-762fc37473f9-konnectivity-ca\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119375    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-bin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119389    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-kubelet\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119431    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-log-socket\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119480    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-config\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.120614 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119499    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119519    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-kubelet-config\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119533    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-k8s-cni-cncf-io\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119558    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-conf-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119585    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-bin\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119602    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zz6\" (UniqueName: \"kubernetes.io/projected/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-kube-api-access-r7zz6\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119616    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4pc\" (UniqueName: \"kubernetes.io/projected/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-kube-api-access-8p4pc\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119640    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-system-cni-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.121376 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.119660    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cnibin\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.135975 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.135932    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"78bd4376b5e50178846f4b8cc0416cb3","Type":"ContainerStarted","Data":"3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d"}
Oct 31 16:01:57.136962 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.136944    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"29bfc3859f40051644e6484149c81648","Type":"ContainerStarted","Data":"219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4"}
Oct 31 16:01:57.138484 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.138450    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 16:01:57.138588 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.138571    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 16:01:57.138632 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.138571    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wg2f6\""
Oct 31 16:01:57.139387 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.139373    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 16:01:57.139502 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.139436    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnhm9\""
Oct 31 16:01:57.139564 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.139551    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.139603 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.139563    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 16:01:57.139686 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.139602    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.143575 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.143558    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnjtb\""
Oct 31 16:01:57.143665 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.143643    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 16:01:57.143726 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.143704    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.143778 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.143737    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.145616 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.145601    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.145701 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.145603    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 16:01:57.145737 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.145719    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.145767 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.145756    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d4rvf\""
Oct 31 16:01:57.147544 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.147529    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.206349 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.206316    2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 16:01:57.220173 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220154    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-systemd-units\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.220226 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220181    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.220226 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220198    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-system-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.220226 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220218    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-netns\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.220323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220235    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-socket-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.220323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220253    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.220323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220253    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-systemd-units\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.220323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220277    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/b2beb745-0c6b-4060-86fd-0e31575d1342-kube-api-access-vzvqg\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.220323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220301    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-daemon-config\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.220323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220311    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-netns\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.220319    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220353    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-system-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220323    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-env-overrides\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.220395    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret podName:28de26f0-048d-48c5-9df2-d1c4c33f4fb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:57.720370466 +0000 UTC m=+3.067727796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret") pod "global-pull-secret-syncer-5n9tx" (UID: "28de26f0-048d-48c5-9df2-d1c4c33f4fb2") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220396    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-socket-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220390    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220426    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88955cf1-7c74-4198-a5a6-762fc37473f9-konnectivity-ca\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220467    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-bin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220512    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-kubelet\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220538    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-log-socket\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220569    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-config\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220587    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.220605 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220604    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-kubelet-config\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220627    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-k8s-cni-cncf-io\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220570    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-kubelet\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220652    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-conf-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220588    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-bin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220663    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-log-socket\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220677    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-bin\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220712    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-kubelet-config\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220717    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-conf-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220748    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220755    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-k8s-cni-cncf-io\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220777    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zz6\" (UniqueName: \"kubernetes.io/projected/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-kube-api-access-r7zz6\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220790    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-bin\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220800    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4pc\" (UniqueName: \"kubernetes.io/projected/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-kube-api-access-8p4pc\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220816    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-system-cni-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220842    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cnibin\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220868    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221187 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220873    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-system-cni-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220885    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cnibin\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220890    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88955cf1-7c74-4198-a5a6-762fc37473f9-agent-certs\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220932    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220960    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cnibin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220973    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88955cf1-7c74-4198-a5a6-762fc37473f9-konnectivity-ca\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220983    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.220982    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-daemon-config\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221010    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhms\" (UniqueName: \"kubernetes.io/projected/251fef43-4a22-4508-9e24-166a266d95a0-kube-api-access-svhms\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221018    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cnibin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221044    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-host-slash\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221079    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-dbus\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221086    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221183    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-conf\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221212    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-lib-modules\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221251    2587 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221278    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-lib-modules\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221287    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-dbus\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.221967 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221331    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-conf\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221357    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-script-lib\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221382    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221398    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-kubelet\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221414    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221429    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-device-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221435    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221472    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-kubelet\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221394    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221490    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-device-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221514    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-slash\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221530    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd2188c-94c4-4700-b0ff-9b7cf279911c-host\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221547    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-tmp\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221557    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-slash\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221575    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-var-lib-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221582    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd2188c-94c4-4700-b0ff-9b7cf279911c-host\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221103    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-host-slash\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.222883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221604    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221629    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-var-lib-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221630    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221671    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da23b925-afa4-4246-a9f9-38b52246d95a-hosts-file\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221695    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cni-binary-copy\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221720    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-etc-kubernetes\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221721    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da23b925-afa4-4246-a9f9-38b52246d95a-hosts-file\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221762    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-etc-kubernetes\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221778    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-registration-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221745    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-registration-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221811    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-sys-fs\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221834    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-kubernetes\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221858    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-sys-fs\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221859    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221879    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-kubernetes\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221881    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221907    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4x8\" (UniqueName: \"kubernetes.io/projected/bdd2188c-94c4-4700-b0ff-9b7cf279911c-kube-api-access-dg4x8\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.223415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221931    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-sys\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221956    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-etc-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.221985    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rbk\" (UniqueName: \"kubernetes.io/projected/9e659030-d532-4842-8462-7b89281cd2f5-kube-api-access-c2rbk\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222021    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-multus\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222035    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-etc-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222045    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-hostroot\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222079    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-multus-certs\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222110    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222127    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-multus\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222127    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cni-binary-copy\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222143    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysconfig\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222152    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-multus-certs\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222167    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-tuned\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222177    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysconfig\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222173    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222196    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-systemd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222216    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-systemd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222220    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.223927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222143    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-hostroot\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222244    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222248    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd2188c-94c4-4700-b0ff-9b7cf279911c-serviceca\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222274    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-modprobe-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222297    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-run\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222320    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-netd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222340    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-run\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222367    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-modprobe-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222371    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-netd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222344    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222415    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzk2\" (UniqueName: \"kubernetes.io/projected/68e7747c-9ba3-420e-aba4-25f0308c6a1e-kube-api-access-9xzk2\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222430    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-sys\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222437    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da23b925-afa4-4246-a9f9-38b52246d95a-tmp-dir\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222493    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-var-lib-kubelet\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222516    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-node-log\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222539    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-iptables-alerter-script\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222565    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-socket-dir-parent\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222591    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-host\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224446 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222590    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-var-lib-kubelet\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222667    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-node-log\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222675    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-host\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222679    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da23b925-afa4-4246-a9f9-38b52246d95a-tmp-dir\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222702    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-netns\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222729    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-ovn\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222738    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-netns\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222754    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dcn\" (UniqueName: \"kubernetes.io/projected/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-kube-api-access-x5dcn\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222768    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-ovn\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222780    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqcf\" (UniqueName: \"kubernetes.io/projected/da23b925-afa4-4246-a9f9-38b52246d95a-kube-api-access-ghqcf\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222804    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-systemd\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222846    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222927    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.222954    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-systemd\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.223001    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-socket-dir-parent\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.223120    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-iptables-alerter-script\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.224595    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-tuned\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.224626    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-tmp\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.224968 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.224755    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88955cf1-7c74-4198-a5a6-762fc37473f9-agent-certs\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.234761 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.234745    2587 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 16:01:57.234823 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.234779    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pmrfr"
Oct 31 16:01:57.239113 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239099    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 16:01:57.239235 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239214    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f9xg5\""
Oct 31 16:01:57.239409 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239390    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239412    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239437    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239444    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239446    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239441    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tcbkl\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239466    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239474    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgxcq\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239479    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239493    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239511    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 16:01:57.239557 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239580    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 16:01:57.239961 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239645    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msflc\""
Oct 31 16:01:57.239961 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239665    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 16:01:57.239961 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239690    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9wrg2\""
Oct 31 16:01:57.239961 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239704    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 16:01:57.239961 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.239769    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 16:01:57.241612 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.241594    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-env-overrides\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.241681 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.241601    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-config\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.242017 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.241992    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.242308 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.242293    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.242351 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.242300    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-script-lib\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.242569 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.242555    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd2188c-94c4-4700-b0ff-9b7cf279911c-serviceca\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.244188 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.244173    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.275970 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.275948    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:01:57.275970 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.275970    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:01:57.276087 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.275984    2587 projected.go:196] Error preparing data for projected volume kube-api-access-dlw7q for pod openshift-network-diagnostics/network-check-target-cfrr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:57.276087 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.276053    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q podName:3cb7154b-15e9-473c-8091-fd02a0eb5789 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:57.776034218 +0000 UTC m=+3.123391543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dlw7q" (UniqueName: "kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q") pod "network-check-target-cfrr7" (UID: "3cb7154b-15e9-473c-8091-fd02a0eb5789") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:57.277031 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.277004    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/b2beb745-0c6b-4060-86fd-0e31575d1342-kube-api-access-vzvqg\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.278049 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.278032    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zz6\" (UniqueName: \"kubernetes.io/projected/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-kube-api-access-r7zz6\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.278133 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.278067    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4pc\" (UniqueName: \"kubernetes.io/projected/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-kube-api-access-8p4pc\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.278634 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.278608    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhms\" (UniqueName: \"kubernetes.io/projected/251fef43-4a22-4508-9e24-166a266d95a0-kube-api-access-svhms\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.278906 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.278884    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dcn\" (UniqueName: \"kubernetes.io/projected/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-kube-api-access-x5dcn\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.279144 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.279127    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4x8\" (UniqueName: \"kubernetes.io/projected/bdd2188c-94c4-4700-b0ff-9b7cf279911c-kube-api-access-dg4x8\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.279681 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.279665    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzk2\" (UniqueName: \"kubernetes.io/projected/68e7747c-9ba3-420e-aba4-25f0308c6a1e-kube-api-access-9xzk2\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.279849 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.279832    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghqcf\" (UniqueName: \"kubernetes.io/projected/da23b925-afa4-4246-a9f9-38b52246d95a-kube-api-access-ghqcf\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.323143 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.323115    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.323275 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.323158    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rbk\" (UniqueName: \"kubernetes.io/projected/9e659030-d532-4842-8462-7b89281cd2f5-kube-api-access-c2rbk\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.323275 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.323267    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:57.323342 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.323321    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:57.823306831 +0000 UTC m=+3.170664141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:57.379436 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.379373    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rbk\" (UniqueName: \"kubernetes.io/projected/9e659030-d532-4842-8462-7b89281cd2f5-kube-api-access-c2rbk\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.397155 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.397126    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:01:57.402039 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.402024    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gk2gp"
Oct 31 16:01:57.406786 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.406759    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88955cf1_7c74_4198_a5a6_762fc37473f9.slice/crio-0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797 WatchSource:0}: Error finding container 0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797: Status 404 returned error can't find the container with id 0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797
Oct 31 16:01:57.410021 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.409999    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:01:57.410147 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.410121    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc628c2_0e1f_45c4_89f9_c7a85ea04468.slice/crio-919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940 WatchSource:0}: Error finding container 919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940: Status 404 returned error can't find the container with id 919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940
Oct 31 16:01:57.418114 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.418093    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cad6ef_ae99_4477_bd4c_e673dd6791ff.slice/crio-4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb WatchSource:0}: Error finding container 4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb: Status 404 returned error can't find the container with id 4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb
Oct 31 16:01:57.419680 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.419664    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:01:57.422903 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.422887    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:01:57.429734 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.429713    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251fef43_4a22_4508_9e24_166a266d95a0.slice/crio-0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d WatchSource:0}: Error finding container 0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d: Status 404 returned error can't find the container with id 0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d
Oct 31 16:01:57.430234 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.430214    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2beb745_0c6b_4060_86fd_0e31575d1342.slice/crio-785741680c0d751e9336937e8f304c9116163187a2e06703cc68e04fa0ab286d WatchSource:0}: Error finding container 785741680c0d751e9336937e8f304c9116163187a2e06703cc68e04fa0ab286d: Status 404 returned error can't find the container with id 785741680c0d751e9336937e8f304c9116163187a2e06703cc68e04fa0ab286d
Oct 31 16:01:57.439392 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.439377    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:01:57.445116 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.445100    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:01:57.448260 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.448243    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda23b925_afa4_4246_a9f9_38b52246d95a.slice/crio-3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74 WatchSource:0}: Error finding container 3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74: Status 404 returned error can't find the container with id 3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74
Oct 31 16:01:57.452071 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.452055    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:01:57.453223 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.453203    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd2188c_94c4_4700_b0ff_9b7cf279911c.slice/crio-00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c WatchSource:0}: Error finding container 00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c: Status 404 returned error can't find the container with id 00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c
Oct 31 16:01:57.460134 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.460119    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e7747c_9ba3_420e_aba4_25f0308c6a1e.slice/crio-5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248 WatchSource:0}: Error finding container 5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248: Status 404 returned error can't find the container with id 5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248
Oct 31 16:01:57.460891 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.460877    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:01:57.469349 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:01:57.469327    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e69cb7a_20ad_47dd_8a50_c5a6a3299fd7.slice/crio-1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306 WatchSource:0}: Error finding container 1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306: Status 404 returned error can't find the container with id 1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306
Oct 31 16:01:57.725810 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.725777    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:57.725953 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.725929    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:57.726018 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.726005    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret podName:28de26f0-048d-48c5-9df2-d1c4c33f4fb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:58.72598001 +0000 UTC m=+4.073337321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret") pod "global-pull-secret-syncer-5n9tx" (UID: "28de26f0-048d-48c5-9df2-d1c4c33f4fb2") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:57.826467 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.826421    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:01:57.826650 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:57.826513    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:57.826650 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.826618    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:57.826741 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.826664    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:58.826651226 +0000 UTC m=+4.174008532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:57.826741 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.826723    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:01:57.826741 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.826736    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:01:57.826888 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.826748    2587 projected.go:196] Error preparing data for projected volume kube-api-access-dlw7q for pod openshift-network-diagnostics/network-check-target-cfrr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:57.826888 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:57.826777    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q podName:3cb7154b-15e9-473c-8091-fd02a0eb5789 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:58.826770135 +0000 UTC m=+4.174127441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlw7q" (UniqueName: "kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q") pod "network-check-target-cfrr7" (UID: "3cb7154b-15e9-473c-8091-fd02a0eb5789") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:58.139921 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.139837    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248"}
Oct 31 16:01:58.141518 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.141491    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c6ssr" event={"ID":"da23b925-afa4-4246-a9f9-38b52246d95a","Type":"ContainerStarted","Data":"3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74"}
Oct 31 16:01:58.143083 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.143057    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fr6l" event={"ID":"d7cad6ef-ae99-4477-bd4c-e673dd6791ff","Type":"ContainerStarted","Data":"4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb"}
Oct 31 16:01:58.144915 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.144893    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk2gp" event={"ID":"5bc628c2-0e1f-45c4-89f9-c7a85ea04468","Type":"ContainerStarted","Data":"919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940"}
Oct 31 16:01:58.146232 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.146209    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cg6j2" event={"ID":"bdd2188c-94c4-4700-b0ff-9b7cf279911c","Type":"ContainerStarted","Data":"00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c"}
Oct 31 16:01:58.147327 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.147304    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vv5k4" event={"ID":"b2beb745-0c6b-4060-86fd-0e31575d1342","Type":"ContainerStarted","Data":"785741680c0d751e9336937e8f304c9116163187a2e06703cc68e04fa0ab286d"}
Oct 31 16:01:58.148664 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.148644    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk" event={"ID":"251fef43-4a22-4508-9e24-166a266d95a0","Type":"ContainerStarted","Data":"0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d"}
Oct 31 16:01:58.149816 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.149794    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b4fr7" event={"ID":"88955cf1-7c74-4198-a5a6-762fc37473f9","Type":"ContainerStarted","Data":"0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797"}
Oct 31 16:01:58.151505 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.151478    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerStarted","Data":"1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306"}
Oct 31 16:01:58.239061 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.239020    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:57 +0000 UTC" deadline="2028-06-15 18:46:15.75763257 +0000 UTC"
Oct 31 16:01:58.239061 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.239058    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14234h44m17.518578991s"
Oct 31 16:01:58.734373 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.734335    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:58.734580 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.734487    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:58.734580 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.734548    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret podName:28de26f0-048d-48c5-9df2-d1c4c33f4fb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:00.734530124 +0000 UTC m=+6.081887435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret") pod "global-pull-secret-syncer-5n9tx" (UID: "28de26f0-048d-48c5-9df2-d1c4c33f4fb2") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:58.835617 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.835576    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:58.835775 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:58.835629    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:01:58.835835 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.835790    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:01:58.835835 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.835809    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:01:58.835835 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.835822    2587 projected.go:196] Error preparing data for projected volume kube-api-access-dlw7q for pod openshift-network-diagnostics/network-check-target-cfrr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:58.835980 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.835876    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q podName:3cb7154b-15e9-473c-8091-fd02a0eb5789 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:00.835858281 +0000 UTC m=+6.183215587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlw7q" (UniqueName: "kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q") pod "network-check-target-cfrr7" (UID: "3cb7154b-15e9-473c-8091-fd02a0eb5789") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:58.835980 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.835946    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:58.836084 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:58.835981    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:00.835969575 +0000 UTC m=+6.183326884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:59.134077 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:59.133970    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:01:59.134077 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:59.134031    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:01:59.134323 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:59.133974    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:01:59.134323 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:59.134142    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:01:59.134323 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:59.134214    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:01:59.134323 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:01:59.134263    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:01:59.239825 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:59.239762    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:57 +0000 UTC" deadline="2028-04-10 12:57:00.213083842 +0000 UTC"
Oct 31 16:01:59.239825 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:01:59.239822    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12644h55m0.973266322s"
Oct 31 16:02:00.754139 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:00.753858    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:00.754139 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.754054    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:00.754139 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.754112    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret podName:28de26f0-048d-48c5-9df2-d1c4c33f4fb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:04.75409346 +0000 UTC m=+10.101450785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret") pod "global-pull-secret-syncer-5n9tx" (UID: "28de26f0-048d-48c5-9df2-d1c4c33f4fb2") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:00.854659    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:00.854719    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.854806    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.854851    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:04.854838656 +0000 UTC m=+10.202195962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.854907    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.854916    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.854924    2587 projected.go:196] Error preparing data for projected volume kube-api-access-dlw7q for pod openshift-network-diagnostics/network-check-target-cfrr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:00.855213 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:00.854948    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q podName:3cb7154b-15e9-473c-8091-fd02a0eb5789 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:04.854940657 +0000 UTC m=+10.202297962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlw7q" (UniqueName: "kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q") pod "network-check-target-cfrr7" (UID: "3cb7154b-15e9-473c-8091-fd02a0eb5789") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:01.135850 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:01.135367    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:01.135850 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:01.135519    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:01.135850 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:01.135591    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:01.135850 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:01.135671    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:01.135850 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:01.135715    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:01.135850 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:01.135783    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:01.167653 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:01.167620    2587 generic.go:356] "Generic (PLEG): container finished" podID="78bd4376b5e50178846f4b8cc0416cb3" containerID="ab45be8c5f59eae8ee3ce1dc720a7bdff4874beb01e0731609e873cb00ee83ed" exitCode=0
Oct 31 16:02:01.167777 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:01.167714    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"78bd4376b5e50178846f4b8cc0416cb3","Type":"ContainerDied","Data":"ab45be8c5f59eae8ee3ce1dc720a7bdff4874beb01e0731609e873cb00ee83ed"}
Oct 31 16:02:01.182743 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:01.181027    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"29bfc3859f40051644e6484149c81648","Type":"ContainerStarted","Data":"6d3842dea956b1395a7d811933b46447f12e305e16bde49c659f1a5ee8e0d8c5"}
Oct 31 16:02:02.203383 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:02.202612    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="0e56a70b3206c203fa512305f14d6016e69abb81dcb4e3d93e44e768b7836ebb" exitCode=0
Oct 31 16:02:02.203383 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:02.202726    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerDied","Data":"0e56a70b3206c203fa512305f14d6016e69abb81dcb4e3d93e44e768b7836ebb"}
Oct 31 16:02:02.212392 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:02.212071    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"78bd4376b5e50178846f4b8cc0416cb3","Type":"ContainerStarted","Data":"25fb02465d55da832bcacc416d091619f790ca3b466171a3c5f5c13fb724b548"}
Oct 31 16:02:02.420778 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:02.419479    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal" podStartSLOduration=6.41944508 podStartE2EDuration="6.41944508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:01.398648514 +0000 UTC m=+6.746005840" watchObservedRunningTime="2026-10-31 16:02:02.41944508 +0000 UTC m=+7.766802410"
Oct 31 16:02:02.484709 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:02.484644    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" podStartSLOduration=6.484624216 podStartE2EDuration="6.484624216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:02.484326914 +0000 UTC m=+7.831684244" watchObservedRunningTime="2026-10-31 16:02:02.484624216 +0000 UTC m=+7.831981545"
Oct 31 16:02:03.135083 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:03.132622    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:03.135083 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:03.132764    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:03.135083 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:03.134886    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:03.135083 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:03.134969    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:03.135083 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:03.134996    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:03.135083 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:03.135041    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:03.217760 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:03.217680    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b4fr7" event={"ID":"88955cf1-7c74-4198-a5a6-762fc37473f9","Type":"ContainerStarted","Data":"7636485755b052c77e404b067a6092ef8f7060cab5685fb52cff3dd515e2157f"}
Oct 31 16:02:03.268949 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:03.268912    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:04.134749 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.134720    2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:04.135415 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.135393    2587 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:04.228964 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.228929    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cg6j2" event={"ID":"bdd2188c-94c4-4700-b0ff-9b7cf279911c","Type":"ContainerStarted","Data":"ec1adeba470af3bcfc9662f76152a55dbc5c790f9ee597222dcbf65e944ef9eb"}
Oct 31 16:02:04.232259 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.232233    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk" event={"ID":"251fef43-4a22-4508-9e24-166a266d95a0","Type":"ContainerStarted","Data":"3de0989928ce09a5e7ef803b51dfb6239bbf59df2ec5c4f86d998e11dccc0de5"}
Oct 31 16:02:04.232794 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.232762    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:04.251423 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.251369    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b4fr7" podStartSLOduration=4.6204130679999995 podStartE2EDuration="9.251348507s" totalImagesPullingTime="4.630935439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:03.330267779 +0000 UTC m=+8.677625108" watchObservedRunningTime="2026-10-31 16:02:04.251348507 +0000 UTC m=+9.598705835"
Oct 31 16:02:04.787676 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.787640    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:04.787843 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.787799    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:04.787919 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.787860    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret podName:28de26f0-048d-48c5-9df2-d1c4c33f4fb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:12.787842202 +0000 UTC m=+18.135199511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret") pod "global-pull-secret-syncer-5n9tx" (UID: "28de26f0-048d-48c5-9df2-d1c4c33f4fb2") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:04.889021 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.888976    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:04.889021 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:04.889022    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:04.889240 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.889161    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:02:04.889240 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.889178    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:02:04.889240 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.889199    2587 projected.go:196] Error preparing data for projected volume kube-api-access-dlw7q for pod openshift-network-diagnostics/network-check-target-cfrr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:04.889345 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.889251    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q podName:3cb7154b-15e9-473c-8091-fd02a0eb5789 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:12.88923168 +0000 UTC m=+18.236588991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlw7q" (UniqueName: "kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q") pod "network-check-target-cfrr7" (UID: "3cb7154b-15e9-473c-8091-fd02a0eb5789") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:04.889438 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.889348    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:04.889438 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:04.889390    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:12.88937826 +0000 UTC m=+18.236735581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:05.135310 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:05.135244    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:05.135447 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:05.135333    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:05.135447 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:05.135407    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:05.135562 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:05.135477    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:05.135562 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:05.135520    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:05.135673 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:05.135586    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:05.998166 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:05.998137    2587 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 16:02:06.183315 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:06.183137    2587 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T16:02:05.998161254Z","UUID":"70417dcd-cf02-4978-8ea0-60b54895b42e","Handler":null,"Name":"","Endpoint":""}
Oct 31 16:02:06.185481 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:06.185444    2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 16:02:06.185604 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:06.185489    2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 16:02:06.238188 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:06.238152    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gk2gp" event={"ID":"5bc628c2-0e1f-45c4-89f9-c7a85ea04468","Type":"ContainerStarted","Data":"985da99fcb3ec1b9e0dec66f43efe8e943efc2652386b10b19ed789e82e1075d"}
Oct 31 16:02:06.241571 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:06.241483    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk" event={"ID":"251fef43-4a22-4508-9e24-166a266d95a0","Type":"ContainerStarted","Data":"df97b55a755e34ccdbdb8617558ae7752c223618a4d2107c35ce64376e2fe781"}
Oct 31 16:02:06.353989 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:06.353849    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cg6j2" podStartSLOduration=5.660850576 podStartE2EDuration="11.353829177s" totalImagesPullingTime="5.692978601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:04.475948017 +0000 UTC m=+9.823305344" watchObservedRunningTime="2026-10-31 16:02:06.353829177 +0000 UTC m=+11.701186518"
Oct 31 16:02:07.133661 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.133605    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:07.133661 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.133618    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:07.134221 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.133768    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:07.134221 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:07.133775    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:07.134221 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:07.133880    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:07.134221 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:07.133958    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:07.246970 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.246931    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="879bf3c730502b12b4adc7fa2a8e0d4827b38e575f767335e415df83d2e4e703" exitCode=0
Oct 31 16:02:07.247125 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.247027    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerDied","Data":"879bf3c730502b12b4adc7fa2a8e0d4827b38e575f767335e415df83d2e4e703"}
Oct 31 16:02:07.252996 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.252973    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c6ssr" event={"ID":"da23b925-afa4-4246-a9f9-38b52246d95a","Type":"ContainerStarted","Data":"b4c35a4e8f247a8076757fc1d9f00ef589700551174f976f22e912c6d0af8011"}
Oct 31 16:02:07.380029 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.379963    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gk2gp" podStartSLOduration=4.395023196 podStartE2EDuration="12.379945532s" totalImagesPullingTime="7.984922336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:06.355045542 +0000 UTC m=+11.702402872" watchObservedRunningTime="2026-10-31 16:02:07.379945532 +0000 UTC m=+12.727302862"
Oct 31 16:02:07.493830 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:07.493783    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c6ssr" podStartSLOduration=3.751726877 podStartE2EDuration="12.493769813s" totalImagesPullingTime="8.742042936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:07.493574731 +0000 UTC m=+12.840932060" watchObservedRunningTime="2026-10-31 16:02:07.493769813 +0000 UTC m=+12.841127141"
Oct 31 16:02:08.257186 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:08.257150    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vv5k4" event={"ID":"b2beb745-0c6b-4060-86fd-0e31575d1342","Type":"ContainerStarted","Data":"ac9cf3558805a6469793249959e50fb527c86a9d3b30ae855f5d5bd156a96a43"}
Oct 31 16:02:08.259500 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:08.259467    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk" event={"ID":"251fef43-4a22-4508-9e24-166a266d95a0","Type":"ContainerStarted","Data":"9e69ee50923289fed5cd786104afc2c0b1ee6136e6d3c7d337257bb233fbd78b"}
Oct 31 16:02:08.261008 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:08.260924    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2fr6l" event={"ID":"d7cad6ef-ae99-4477-bd4c-e673dd6791ff","Type":"ContainerStarted","Data":"a85bd0fe68967445817a5462f986fd15e06c699b6b108523668341333546555f"}
Oct 31 16:02:08.371829 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:08.371782    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vv5k4" podStartSLOduration=3.005403459 podStartE2EDuration="13.371765954s" totalImagesPullingTime="10.366362495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:08.371588675 +0000 UTC m=+13.718946003" watchObservedRunningTime="2026-10-31 16:02:08.371765954 +0000 UTC m=+13.719123282"
Oct 31 16:02:08.484144 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:08.484098    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2fr6l" podStartSLOduration=4.981828852 podStartE2EDuration="13.484083829s" totalImagesPullingTime="8.502254977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:08.483568634 +0000 UTC m=+13.830925962" watchObservedRunningTime="2026-10-31 16:02:08.484083829 +0000 UTC m=+13.831441156"
Oct 31 16:02:08.597492 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:08.597393    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk" podStartSLOduration=3.828782381 podStartE2EDuration="13.597377163s" totalImagesPullingTime="9.768594782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:08.597066189 +0000 UTC m=+13.944423518" watchObservedRunningTime="2026-10-31 16:02:08.597377163 +0000 UTC m=+13.944734491"
Oct 31 16:02:09.131890 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:09.131860    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:09.132049 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:09.131872    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:09.132049 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:09.131986    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:09.132165 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:09.132082    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:09.132165 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:09.132119    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:09.132257 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:09.132182    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:09.265771 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:09.265734    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="22e5fe6d5fa293d1d3a9a3bf623563dcc7146900c35f3658562ae9b2f3ca7ba2" exitCode=0
Oct 31 16:02:09.266408 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:09.266034    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerDied","Data":"22e5fe6d5fa293d1d3a9a3bf623563dcc7146900c35f3658562ae9b2f3ca7ba2"}
Oct 31 16:02:11.132551 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:11.132522    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:11.133152 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:11.132557    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:11.133152 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:11.132647    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:11.133152 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:11.132738    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:11.133320 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:11.133303    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:11.133414 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:11.133393    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:11.273419 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:11.273347    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="0d623ccd0832a8bfb5be1f8d7680b4ee25e85a3a5e7253a6f1849dafc675aaf4" exitCode=0
Oct 31 16:02:11.273419 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:11.273391    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerDied","Data":"0d623ccd0832a8bfb5be1f8d7680b4ee25e85a3a5e7253a6f1849dafc675aaf4"}
Oct 31 16:02:12.870428 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:12.870269    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:12.870914 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.870416    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:12.870914 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.870533    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret podName:28de26f0-048d-48c5-9df2-d1c4c33f4fb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.870512266 +0000 UTC m=+34.217869575 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret") pod "global-pull-secret-syncer-5n9tx" (UID: "28de26f0-048d-48c5-9df2-d1c4c33f4fb2") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:12.971303 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:12.971274    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:12.971403 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:12.971326    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:12.971473 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.971439    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:12.971540 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.971522    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:02:12.971630 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.971549    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:02:12.971630 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.971563    2587 projected.go:196] Error preparing data for projected volume kube-api-access-dlw7q for pod openshift-network-diagnostics/network-check-target-cfrr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:12.971630 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.971529    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.971507273 +0000 UTC m=+34.318864601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:12.971630 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:12.971621    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q podName:3cb7154b-15e9-473c-8091-fd02a0eb5789 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.971606745 +0000 UTC m=+34.318964059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlw7q" (UniqueName: "kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q") pod "network-check-target-cfrr7" (UID: "3cb7154b-15e9-473c-8091-fd02a0eb5789") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:13.132473 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.132221    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:13.132623 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:13.132590    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:13.132623 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.132262    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:13.132706 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:13.132696    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:13.132739 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.132247    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:13.132789 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:13.132772    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:13.280896 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.280862    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"a3618f7a968722e259b3c092df7cf7c3a1598dc7e921838f6be1d492e18139ca"}
Oct 31 16:02:13.281052 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.280953    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"d107741fe2076a2e20819f787ef232525532c1af456b139b588477204280fffc"}
Oct 31 16:02:13.281052 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.281001    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"ea15b44e38ae1b7707729beaf5bd3a13ff80360863631883318bcadfa2ed58c4"}
Oct 31 16:02:13.281052 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.281046    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"109583c709f5a163a43ba5f1fd70ce0674b28944ff0b7b617171d297da18e26f"}
Oct 31 16:02:13.281193 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.281090    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"eb75d373ba4561197549998a32a8c369a3192712d4471d980e6dab31e10967b9"}
Oct 31 16:02:13.281193 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:13.281137    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"d675991a87d4249ac90f494e4ce608489a9b83a17129c34134297635a4b54b7d"}
Oct 31 16:02:14.284943 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:14.284909    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="d6ffdfc95a79a7d1796aaf91ae27b6a05027cbe9a34bb382f312aed2ad718070" exitCode=0
Oct 31 16:02:14.285314 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:14.284959    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerDied","Data":"d6ffdfc95a79a7d1796aaf91ae27b6a05027cbe9a34bb382f312aed2ad718070"}
Oct 31 16:02:15.133582 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:15.133560    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:15.133690 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:15.133672    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:15.133764 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:15.133745    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:15.133942 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:15.133912    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:15.133979 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:15.133952    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:15.134060 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:15.134036    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:15.289962 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:15.289931    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="779fc7c234848d9a8a7effa96b5b81e00c205d2d2338699537dd26d7f1f365e6" exitCode=0
Oct 31 16:02:15.290538 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:15.290007    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerDied","Data":"779fc7c234848d9a8a7effa96b5b81e00c205d2d2338699537dd26d7f1f365e6"}
Oct 31 16:02:15.293142 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:15.293123    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"c5e9abae6d8fa004d6dc9c9633910491e60af31643656f6df443a613867b6d88"}
Oct 31 16:02:16.298600 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:16.298566    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" event={"ID":"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7","Type":"ContainerStarted","Data":"64445487022bebfbe1ec898828b1f81f6bab4ba7afdd25c39bf0d0edd7fb3f4d"}
Oct 31 16:02:16.414379 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:16.414311    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qhr6p" podStartSLOduration=8.025653173 podStartE2EDuration="21.414297707s" totalImagesPullingTime="13.388644534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:16.414105445 +0000 UTC m=+21.761462772" watchObservedRunningTime="2026-10-31 16:02:16.414297707 +0000 UTC m=+21.761655031"
Oct 31 16:02:17.131739 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:17.131709    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:17.131923 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:17.131776    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:17.131923 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:17.131786    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:17.131923 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:17.131873    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:17.132073 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:17.132039    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:17.132148 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:17.132130    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:17.305324 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:17.305294    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" event={"ID":"68e7747c-9ba3-420e-aba4-25f0308c6a1e","Type":"ContainerStarted","Data":"e7863343e5dcf4f82be80a2aa72b8914ac823705e1e773f3bd4c392507c4869d"}
Oct 31 16:02:17.574617 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:17.574559    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8" podStartSLOduration=7.387516807 podStartE2EDuration="22.574546481s" totalImagesPullingTime="15.187029674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:17.524590014 +0000 UTC m=+22.871947341" watchObservedRunningTime="2026-10-31 16:02:17.574546481 +0000 UTC m=+22.921903808"
Oct 31 16:02:18.307751 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:18.307723    2587 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 16:02:18.308219 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:18.308193    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:18.308348 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:18.308269    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:18.323197 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:18.323176    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:18.323394 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:18.323378    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:18.533410 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:18.533383    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:19.131989 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.131956    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:19.131989 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.131976    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:19.132305 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:19.132068    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:19.132305 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:19.132117    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:19.132305 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.132161    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:19.132305 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:19.132239    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:19.883906 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.883716    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5n9tx"]
Oct 31 16:02:19.884321 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.883984    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:19.884321 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:19.884073    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:19.886383 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.886362    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cfrr7"]
Oct 31 16:02:19.886493 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.886449    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:19.886580 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:19.886561    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:19.887017 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.887000    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6rw9"]
Oct 31 16:02:19.887080 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:19.887069    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:19.887188 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:19.887174    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:21.132552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:21.132518    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:21.132552 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:21.132553    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:21.133008 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:21.132524    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:21.133008 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:21.132619    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:21.133008 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:21.132715    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:21.133008 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:21.132783    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:23.132326 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:23.132292    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:23.132911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:23.132292    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:23.132911 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:23.132425    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:23.132911 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:23.132296    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:23.132911 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:23.132502    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:23.132911 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:23.132540    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:25.132417 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:25.132391    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:25.132875 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:25.132523    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:25.132875 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:25.132542    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6rw9" podUID="9e659030-d532-4842-8462-7b89281cd2f5"
Oct 31 16:02:25.132875 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:25.132593    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n9tx" podUID="28de26f0-048d-48c5-9df2-d1c4c33f4fb2"
Oct 31 16:02:25.132875 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:25.132603    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:25.132875 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:25.132663    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfrr7" podUID="3cb7154b-15e9-473c-8091-fd02a0eb5789"
Oct 31 16:02:26.422367 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.422294    2587 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeReady"
Oct 31 16:02:26.422740 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.422404    2587 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 16:02:26.534768 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.534740    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nfm25"]
Oct 31 16:02:26.576151 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.576116    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bwjhn"]
Oct 31 16:02:26.576287 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.576237    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.590774 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.590754    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfm25"]
Oct 31 16:02:26.590899 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.590859    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bwjhn"]
Oct 31 16:02:26.590899 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.590871    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:26.628867 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.628846    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 16:02:26.628986 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.628886    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tp4vw\""
Oct 31 16:02:26.628986 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.628891    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 16:02:26.643125 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.643107    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 16:02:26.643182 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.643121    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 16:02:26.643182 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.643132    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 16:02:26.643182 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.643161    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mwvsn\""
Oct 31 16:02:26.679534 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.679491    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1476215c-1e66-46ba-a910-58fb73446da7-tmp-dir\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.679534 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.679521    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2v8v\" (UniqueName: \"kubernetes.io/projected/1476215c-1e66-46ba-a910-58fb73446da7-kube-api-access-z2v8v\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.679627 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.679546    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1476215c-1e66-46ba-a910-58fb73446da7-config-volume\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.679627 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.679561    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.780312 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780284    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1476215c-1e66-46ba-a910-58fb73446da7-tmp-dir\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.780441 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780322    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2v8v\" (UniqueName: \"kubernetes.io/projected/1476215c-1e66-46ba-a910-58fb73446da7-kube-api-access-z2v8v\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.780441 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780346    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-cert\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:26.780577 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780503    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nx8\" (UniqueName: \"kubernetes.io/projected/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-kube-api-access-84nx8\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:26.780645 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780628    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1476215c-1e66-46ba-a910-58fb73446da7-config-volume\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.780700 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780635    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1476215c-1e66-46ba-a910-58fb73446da7-tmp-dir\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.780700 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.780788 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:26.780772    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:26.780833 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:26.780823    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:27.280808078 +0000 UTC m=+32.628165391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:26.781013 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.780996    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1476215c-1e66-46ba-a910-58fb73446da7-config-volume\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.839829 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.839799    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2v8v\" (UniqueName: \"kubernetes.io/projected/1476215c-1e66-46ba-a910-58fb73446da7-kube-api-access-z2v8v\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:26.881621 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.881596    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-cert\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:26.881717 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.881625    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nx8\" (UniqueName: \"kubernetes.io/projected/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-kube-api-access-84nx8\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:26.883883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.883855    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-cert\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:26.938648 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:26.938597    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nx8\" (UniqueName: \"kubernetes.io/projected/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-kube-api-access-84nx8\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:27.132348 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.132315    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:27.132540 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.132352    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:27.132603 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.132538    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:27.185080 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185058    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 16:02:27.185258 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185229    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 16:02:27.185373 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185282    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 16:02:27.185433 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185418    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 16:02:27.185503 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185437    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fd4\""
Oct 31 16:02:27.185546 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185501    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4fqcg\""
Oct 31 16:02:27.185546 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.185535    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jqwtc\""
Oct 31 16:02:27.227078 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.227057    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:27.284329 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.284299    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:27.284511 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:27.284491    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:27.284590 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:27.284578    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.28455377 +0000 UTC m=+33.631911080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:27.649961 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:02:27.649935    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb3b8c3_9810_4e2a_8d51_6861faa35baf.slice/crio-2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7 WatchSource:0}: Error finding container 2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7: Status 404 returned error can't find the container with id 2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7
Oct 31 16:02:27.651218 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:27.651192    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bwjhn"]
Oct 31 16:02:28.291284 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:28.291242    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:28.291483 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:28.291370    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:28.291483 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:28.291434    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.291417777 +0000 UTC m=+35.638775086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:28.328813 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:28.328784    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bwjhn" event={"ID":"9fb3b8c3-9810-4e2a-8d51-6861faa35baf","Type":"ContainerStarted","Data":"2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7"}
Oct 31 16:02:28.896503 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:28.896466    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:28.899092 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:28.899068    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:28.997773 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:28.997736    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:28.997941 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:28.997795    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:28.998065 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:28.998035    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:28.998175 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:28.998149    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:03:00.998128064 +0000 UTC m=+66.345485377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : secret "metrics-daemon-secret" not found
Oct 31 16:02:29.000724 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:29.000697    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:29.020700 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:29.020674    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:29.271355 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:29.271325    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:29.405892 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:02:29.405863    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28de26f0_048d_48c5_9df2_d1c4c33f4fb2.slice/crio-67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad WatchSource:0}: Error finding container 67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad: Status 404 returned error can't find the container with id 67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad
Oct 31 16:02:29.405988 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:29.405913    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5n9tx"]
Oct 31 16:02:29.687840 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:02:29.687814    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb7154b_15e9_473c_8091_fd02a0eb5789.slice/crio-b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020 WatchSource:0}: Error finding container b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020: Status 404 returned error can't find the container with id b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020
Oct 31 16:02:29.688735 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:29.688681    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cfrr7"]
Oct 31 16:02:30.309416 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.309379    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:30.309807 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:30.309543    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:30.309807 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:30.309615    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.30959582 +0000 UTC m=+39.656953131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:30.334742 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.334708    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cfrr7" event={"ID":"3cb7154b-15e9-473c-8091-fd02a0eb5789","Type":"ContainerStarted","Data":"b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020"}
Oct 31 16:02:30.335984 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.335958    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5n9tx" event={"ID":"28de26f0-048d-48c5-9df2-d1c4c33f4fb2","Type":"ContainerStarted","Data":"67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad"}
Oct 31 16:02:30.785520 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.785482    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jc94s"]
Oct 31 16:02:30.806015 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.805979    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.812942 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.812912    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-data-volume\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.813086 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.812951    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-crio-socket\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.813086 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.812974    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.813086 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.813035    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.813086 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.813079    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drv8b\" (UniqueName: \"kubernetes.io/projected/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-api-access-drv8b\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.855470 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.855428    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jc94s"]
Oct 31 16:02:30.858088 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.858060    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 16:02:30.858088 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.858065    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 16:02:30.904669 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.904639    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 16:02:30.905023 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.905002    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7g99\""
Oct 31 16:02:30.909197 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.909174    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 16:02:30.910001 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.909970    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"]
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.913476    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-data-volume\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.913523    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-crio-socket\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.913551    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.913587    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.913606    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drv8b\" (UniqueName: \"kubernetes.io/projected/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-api-access-drv8b\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.914227    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-data-volume\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.914384    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-crio-socket\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:30.914442    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:30.914527    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:31.414511114 +0000 UTC m=+36.761868419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : configmap "kube-rbac-proxy" not found
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:30.914592    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:30.914781 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:30.914615    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:31.414608443 +0000 UTC m=+36.761965749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:30.931750 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.931721    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"]
Oct 31 16:02:30.931855 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.931726    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:30.975002 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.974972    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drv8b\" (UniqueName: \"kubernetes.io/projected/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-api-access-drv8b\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:30.978258 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.978233    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-78cj7-pgttm"]
Oct 31 16:02:30.994371 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:30.994347    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.014737 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.014710    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rjm\" (UniqueName: \"kubernetes.io/projected/f92acc0f-003d-425d-861e-a3a949ea34a5-kube-api-access-x9rjm\") pod \"migrator-66df5d496b-6lznt\" (UID: \"f92acc0f-003d-425d-861e-a3a949ea34a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:31.033095 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.033065    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-78cj7-pgttm"]
Oct 31 16:02:31.070786 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.070718    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lhk9l\""
Oct 31 16:02:31.070930 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.070815    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qmh6f\""
Oct 31 16:02:31.071309 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.070992    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 16:02:31.071309 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.071069    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:31.071309 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.071175    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 16:02:31.115895 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.115860    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.116006 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.115939    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.116006 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.115969    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.116090 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.116022    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rjm\" (UniqueName: \"kubernetes.io/projected/f92acc0f-003d-425d-861e-a3a949ea34a5-kube-api-access-x9rjm\") pod \"migrator-66df5d496b-6lznt\" (UID: \"f92acc0f-003d-425d-861e-a3a949ea34a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:31.116090 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.116059    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.123888 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.123855    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 16:02:31.175643 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.175609    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rjm\" (UniqueName: \"kubernetes.io/projected/f92acc0f-003d-425d-861e-a3a949ea34a5-kube-api-access-x9rjm\") pod \"migrator-66df5d496b-6lznt\" (UID: \"f92acc0f-003d-425d-861e-a3a949ea34a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:31.216540 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.216506    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.216704 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.216605    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.216704 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.216633    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.216704 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.216685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.217111 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.217088    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.217359 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.217328    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.218190 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.218145    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.265114 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.265082    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:31.273390 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.273359    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.334739 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.334632    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:31.419962 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.419608    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:31.419962 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.419661    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:31.419962 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:31.419758    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:31.419962 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:31.419802    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.41979048 +0000 UTC m=+37.767147786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:31.419962 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:31.419835    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 16:02:31.419962 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:31.419857    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.419849043 +0000 UTC m=+37.767206349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : configmap "kube-rbac-proxy" not found
Oct 31 16:02:31.674828 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.674746    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"]
Oct 31 16:02:31.678577 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:02:31.678550    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92acc0f_003d_425d_861e_a3a949ea34a5.slice/crio-f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef WatchSource:0}: Error finding container f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef: Status 404 returned error can't find the container with id f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef
Oct 31 16:02:31.721823 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:31.721799    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-78cj7-pgttm"]
Oct 31 16:02:31.726253 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:02:31.726228    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc498d605_032d_4c7b_82ff_1098b1935a88.slice/crio-f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248 WatchSource:0}: Error finding container f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248: Status 404 returned error can't find the container with id f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248
Oct 31 16:02:32.110130 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.110092    2587 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6wgwt"]
Oct 31 16:02:32.138927 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.138897    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.158216 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.158194    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6wgwt"]
Oct 31 16:02:32.191881 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.191846    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 16:02:32.192082 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.192023    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 16:02:32.192082 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.192036    2587 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 16:02:32.192082 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.192063    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xbg8d\""
Oct 31 16:02:32.192082 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.192047    2587 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 16:02:32.226658 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.226626    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-cabundle\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.226829 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.226667    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvxr\" (UniqueName: \"kubernetes.io/projected/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-kube-api-access-trvxr\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.226829 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.226779    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-key\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.327523 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.327484    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-key\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.327687 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.327616    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-cabundle\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.327687 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.327642    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvxr\" (UniqueName: \"kubernetes.io/projected/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-kube-api-access-trvxr\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.328609 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.328583    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-cabundle\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.330494 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.330473    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-key\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.343018 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.342988    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-78cj7-pgttm" event={"ID":"c498d605-032d-4c7b-82ff-1098b1935a88","Type":"ContainerStarted","Data":"f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248"}
Oct 31 16:02:32.344515 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.344447    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt" event={"ID":"f92acc0f-003d-425d-861e-a3a949ea34a5","Type":"ContainerStarted","Data":"f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef"}
Oct 31 16:02:32.346139 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.346105    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bwjhn" event={"ID":"9fb3b8c3-9810-4e2a-8d51-6861faa35baf","Type":"ContainerStarted","Data":"0da35ac30846eb56e3ee7e718ce0223f32643006d754c6840843a26949082275"}
Oct 31 16:02:32.385883 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.385820    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvxr\" (UniqueName: \"kubernetes.io/projected/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-kube-api-access-trvxr\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.428976 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.428943    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:32.429133 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.429013    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:32.429133 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:32.429111    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 16:02:32.429244 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:32.429183    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.429162704 +0000 UTC m=+39.776520021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : configmap "kube-rbac-proxy" not found
Oct 31 16:02:32.429628 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:32.429607    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:32.429710 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:32.429662    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.429647633 +0000 UTC m=+39.777004940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:32.462306 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.462253    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bwjhn" podStartSLOduration=2.875067959 podStartE2EDuration="6.462234887s" totalImagesPullingTime="3.587166928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:32.461691446 +0000 UTC m=+37.809048775" watchObservedRunningTime="2026-10-31 16:02:32.462234887 +0000 UTC m=+37.809592216"
Oct 31 16:02:32.463768 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.463740    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:32.865780 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:32.865745    2587 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6wgwt"]
Oct 31 16:02:32.866058 ip-10-0-129-180 kubenswrapper[2587]: W1031 16:02:32.865952    2587 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e170cf5_c15f_45bd_9d7d_db33a914f7cc.slice/crio-2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb WatchSource:0}: Error finding container 2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb: Status 404 returned error can't find the container with id 2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb
Oct 31 16:02:33.352794 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:33.352724    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt" event={"ID":"0e170cf5-c15f-45bd-9d7d-db33a914f7cc","Type":"ContainerStarted","Data":"2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb"}
Oct 31 16:02:34.358334 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:34.358126    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt" event={"ID":"f92acc0f-003d-425d-861e-a3a949ea34a5","Type":"ContainerStarted","Data":"cfb55b668bea7aad79d732ba92438b68d2328a5a22d5cdb0d22cd2bd8d6dbeb8"}
Oct 31 16:02:34.358334 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:34.358178    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:34.358334 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:34.358296    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:34.358934 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:34.358353    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:42.358334699 +0000 UTC m=+47.705692009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:34.459522 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:34.459492    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:34.459642 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:34.459564    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:34.459699 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:34.459688    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:34.459752 ip-10-0-129-180 kubenswrapper[2587]: E1031 16:02:34.459740    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.45972237 +0000 UTC m=+43.807079690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:34.460502 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:34.460443    2587 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:34.471134 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:34.471086    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt" podStartSLOduration=2.167277368 podStartE2EDuration="4.471070382s" totalImagesPullingTime="2.303793014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:34.470508044 +0000 UTC m=+39.817865365" watchObservedRunningTime="2026-10-31 16:02:34.471070382 +0000 UTC m=+39.818427711"
Oct 31 16:02:35.362840 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:35.362797    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cfrr7" event={"ID":"3cb7154b-15e9-473c-8091-fd02a0eb5789","Type":"ContainerStarted","Data":"7805a0e6bb89f4d4bf6d361f481196a7d8746c9843c872263662ad37c3050cd8"}
Oct 31 16:02:35.363258 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:35.362904    2587 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:35.364979 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:35.364954    2587 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-78cj7-pgttm" event={"ID":"c498d605-032d-4c7b-82ff-1098b1935a88","Type":"ContainerStarted","Data":"67b11516633f65ce011f9d5476f7f51282a717ac4bd54815aa3601dec867f59c"}
Oct 31 16:02:35.476729 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:35.476678    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cfrr7" podStartSLOduration=35.822178898 podStartE2EDuration="40.476659513s" totalImagesPullingTime="4.654480615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:35.47630744 +0000 UTC m=+40.823664770" watchObservedRunningTime="2026-10-31 16:02:35.476659513 +0000 UTC m=+40.824016842"
Oct 31 16:02:35.592985 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:35.592929    2587 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-78cj7-pgttm" podStartSLOduration=2.200421025 podStartE2EDuration="5.592910446s" totalImagesPullingTime="3.392489421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:35.592417797 +0000 UTC m=+40.939775128" watchObservedRunningTime="2026-10-31 16:02:35.592910446 +0000 UTC m=+40.940267777"
Oct 31 16:02:36.238115 ip-10-0-129-180 kubenswrapper[2587]: I1031 16:02:36.238037    2587 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:02:36.238198 ip-10-0-129-180 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 16:02:36.252848 ip-10-0-129-180 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 16:02:36.253063 ip-10-0-129-180 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 16:02:36.253121 ip-10-0-129-180 systemd[1]: kubelet.service: Consumed 2.344s CPU time.
Oct 31 16:02:36.270100 ip-10-0-129-180 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 16:02:36.328469 ip-10-0-129-180 kubenswrapper[4519]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:02:36.328469 ip-10-0-129-180 kubenswrapper[4519]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 16:02:36.328469 ip-10-0-129-180 kubenswrapper[4519]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:02:36.329025 ip-10-0-129-180 kubenswrapper[4519]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:02:36.330508 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330415    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.330508 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330507    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330515    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330521    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330525    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330528    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330531    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330533    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330536    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330539    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330541    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330544    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330547    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330550    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330553    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330555    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330558    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330560    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330563    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330566    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.330581 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330568    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330571    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330573    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330576    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330579    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330581    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330584    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330594    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330598    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330600    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330603    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330606    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330608    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330611    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330614    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330616    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330619    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330621    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330623    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330626    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.331046 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330628    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330631    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330634    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330636    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330639    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330641    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330644    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330646    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330650    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330653    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330655    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330659    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330662    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330665    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330668    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330670    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330673    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330675    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330678    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.331560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330681    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330684    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330686    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330689    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330692    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330694    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330697    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330699    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330702    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330704    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330707    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330710    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330712    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330715    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330718    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330721    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330723    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330726    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330728    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330731    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.332087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330733    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330737    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330740    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330743    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330746    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330748    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330751    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330754    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330756    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330759    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330761    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330764    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330767    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330770    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330772    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330775    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330778    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330780    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330783    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.332880 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330785    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330788    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330791    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330793    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330796    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330799    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330803    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.330807    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331565    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331573    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331576    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331579    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331583    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331585    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331589    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331592    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331596    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331600    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331603    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.333578 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331606    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331609    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331611    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331614    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331616    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331619    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331623    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331631    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331634    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331637    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331640    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331644    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331646    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331649    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331669    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331674    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331677    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331681    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331685    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.334111 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331688    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331690    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331693    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331696    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331698    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331701    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331703    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331706    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331708    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331711    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331713    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331716    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331718    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331721    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331724    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331726    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331729    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331731    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331735    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331738    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.334600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331741    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331744    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331746    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331749    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331752    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331754    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331756    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331759    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331761    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331764    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331766    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331770    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331772    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331775    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331777    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331780    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331782    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331785    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331787    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.335090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331792    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331795    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331799    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331803    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331807    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331811    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331815    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331819    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331822    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331824    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331827    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331830    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331833    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331836    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331838    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331842    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331845    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331847    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331850    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331853    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.335567 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331856    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331858    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331861    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331863    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331866    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331868    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331871    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331873    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331876    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331879    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331883    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331888    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331890    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331893    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331896    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331899    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.331902    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332503    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332511    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332514    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.336058 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332517    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332520    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332523    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332526    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332528    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332531    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332534    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332537    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332542    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332546    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332551    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332556    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332559    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332562    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332564    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332567    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332570    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332572    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332575    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332578    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.336701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332580    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332583    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332586    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332592    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332595    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332597    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332600    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332602    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332606    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332610    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332614    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332616    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332620    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332627    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332633    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332637    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332640    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332643    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332646    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.337570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332648    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332652    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332654    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332657    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332660    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332662    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332665    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332670    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332673    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332678    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332683    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332687    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332691    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332693    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332696    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332701    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332707    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332711    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332715    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.338376 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332717    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332720    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332723    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332725    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332728    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332731    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332735    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332739    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332742    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332744    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332747    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332750    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332752    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332755    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332757    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332761    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332766    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332768    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332771    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332773    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.339188 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332777    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332781    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332786    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332790    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332792    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332795    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332799    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332803    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332806    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332810    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332814    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332817    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332820    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332822    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332825    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332828    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332830    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332833    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332835    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332838    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.340098 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332840    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332843    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332845    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332848    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.332851    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.332968    4519 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.332977    4519 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.332986    4519 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.332994    4519 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.332998    4519 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333002    4519 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333010    4519 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333015    4519 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333019    4519 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333022    4519 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333025    4519 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333030    4519 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333035    4519 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333040    4519 flags.go:64] FLAG: --cgroup-root=""
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333045    4519 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333048    4519 flags.go:64] FLAG: --client-ca-file=""
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333053    4519 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333056    4519 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 16:02:36.340740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333061    4519 flags.go:64] FLAG: --cluster-domain=""
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333064    4519 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333067    4519 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333070    4519 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333074    4519 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333078    4519 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333081    4519 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333084    4519 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333087    4519 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333090    4519 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333094    4519 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333097    4519 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333100    4519 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333103    4519 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333108    4519 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333114    4519 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333119    4519 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333124    4519 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333128    4519 flags.go:64] FLAG: --enable-server="true"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333131    4519 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333135    4519 flags.go:64] FLAG: --event-burst="100"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333138    4519 flags.go:64] FLAG: --event-qps="50"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333141    4519 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333144    4519 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 16:02:36.341367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333147    4519 flags.go:64] FLAG: --eviction-hard=""
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333152    4519 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333155    4519 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333158    4519 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333161    4519 flags.go:64] FLAG: --eviction-soft=""
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333164    4519 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333168    4519 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333171    4519 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333174    4519 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333177    4519 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333180    4519 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333184    4519 flags.go:64] FLAG: --feature-gates=""
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333190    4519 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333196    4519 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333201    4519 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333205    4519 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333208    4519 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333211    4519 flags.go:64] FLAG: --help="false"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333214    4519 flags.go:64] FLAG: --hostname-override="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333218    4519 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333221    4519 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333224    4519 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333227    4519 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 16:02:36.342116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333231    4519 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333234    4519 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333239    4519 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333241    4519 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333245    4519 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333248    4519 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333251    4519 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333254    4519 flags.go:64] FLAG: --kube-reserved=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333257    4519 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333260    4519 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333264    4519 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333269    4519 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333274    4519 flags.go:64] FLAG: --lock-file=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333278    4519 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333282    4519 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333287    4519 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333292    4519 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333295    4519 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333298    4519 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333301    4519 flags.go:64] FLAG: --logging-format="text"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333304    4519 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333308    4519 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333311    4519 flags.go:64] FLAG: --manifest-url=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333314    4519 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333318    4519 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 16:02:36.342772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333321    4519 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333326    4519 flags.go:64] FLAG: --max-pods="110"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333329    4519 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333332    4519 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333335    4519 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333338    4519 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333342    4519 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333347    4519 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333352    4519 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333361    4519 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333364    4519 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333369    4519 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333374    4519 flags.go:64] FLAG: --pod-cidr=""
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333377    4519 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333380    4519 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333383    4519 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333386    4519 flags.go:64] FLAG: --port="10250"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333389    4519 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333392    4519 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0cc520d52fbe906ec"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333396    4519 flags.go:64] FLAG: --qos-reserved=""
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333399    4519 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333402    4519 flags.go:64] FLAG: --register-node="true"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333407    4519 flags.go:64] FLAG: --register-with-taints=""
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333411    4519 flags.go:64] FLAG: --registry-burst="10"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333414    4519 flags.go:64] FLAG: --registry-qps="5"
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333418    4519 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 16:02:36.343443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333423    4519 flags.go:64] FLAG: --reserved-memory=""
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333430    4519 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333434    4519 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333438    4519 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333441    4519 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333444    4519 flags.go:64] FLAG: --runonce="false"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333447    4519 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333464    4519 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333469    4519 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333472    4519 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333475    4519 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333478    4519 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333482    4519 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333485    4519 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333487    4519 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333492    4519 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333495    4519 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333498    4519 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333501    4519 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333504    4519 flags.go:64] FLAG: --system-cgroups=""
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333507    4519 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333513    4519 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333516    4519 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333519    4519 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333549    4519 flags.go:64] FLAG: --tls-min-version=""
Oct 31 16:02:36.344203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333553    4519 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333556    4519 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333559    4519 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333563    4519 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333566    4519 flags.go:64] FLAG: --v="2"
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333571    4519 flags.go:64] FLAG: --version="false"
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333575    4519 flags.go:64] FLAG: --vmodule=""
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333579    4519 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.333583    4519 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333730    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333735    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333738    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333741    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333744    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333747    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333750    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333752    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333755    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333758    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333761    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333763    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.344998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333766    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333770    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333773    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333776    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333780    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333784    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333788    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333791    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333795    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333798    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333800    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333803    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333805    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333808    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333812    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333814    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333817    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333819    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333822    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.345796 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333825    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333827    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333830    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333832    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333835    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333838    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333841    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333843    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333846    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333848    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333851    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333853    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333856    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333859    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333862    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333865    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333867    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333870    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333872    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333875    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.346315 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333878    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333881    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333884    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333886    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333889    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333892    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333894    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333898    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333901    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333903    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333906    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333909    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333911    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333914    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333916    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333919    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333922    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333924    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333927    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333930    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.346850 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333932    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333936    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333940    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333943    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333946    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333949    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333953    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333956    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333959    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333961    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333964    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333967    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333969    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333972    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333975    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333977    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333980    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333982    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333985    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.347427 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333989    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333992    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.333994    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334003    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334006    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334009    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334012    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334014    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334017    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334019    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334022    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334025    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334028    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334031    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334033    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334036    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.348279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.334064    4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334098    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334103    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334107    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334111    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334113    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334116    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334119    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334122    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334125    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334128    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334131    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334134    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334138    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334141    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334145    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334148    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334152    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334154    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334157    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.348711 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334160    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334163    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334165    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334168    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334170    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334173    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334175    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334178    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334181    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334183    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334186    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334188    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334191    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334193    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334196    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334199    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334202    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334204    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334207    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.349198 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334209    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334212    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334214    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334217    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334219    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334222    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334225    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334228    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334230    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334233    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334237    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334240    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334242    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334245    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334248    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334250    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334253    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334256    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334258    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334261    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.349675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334264    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334266    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334269    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334272    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334274    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334278    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334281    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334284    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334287    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334290    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334293    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334295    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334298    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334300    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334303    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334305    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334308    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334310    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334314    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.350274 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334317    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334319    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334322    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334327    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334330    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334332    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334336    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334339    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334344    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334348    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334362    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334366    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334368    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334371    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334374    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334377    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334380    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334382    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334385    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334387    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.350878 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334390    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334394    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334396    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334399    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334402    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334404    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334407    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334409    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.334412    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.339836    4519 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.339852    4519 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.339870    4519 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.339875    4519 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339945    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339950    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339953    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339956    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339959    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339962    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.351619 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339965    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339968    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339970    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339975    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339978    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339981    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339985    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339990    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339993    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.339996    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340001    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340004    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340007    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340010    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340013    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340016    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340018    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340021    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340024    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.352085 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340027    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340030    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340033    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340036    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340038    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340041    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340044    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340047    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340050    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340052    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340056    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340058    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340061    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340064    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340066    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340069    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340072    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340075    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340077    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340080    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.352574 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340082    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340085    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340088    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340090    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340093    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340096    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340099    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340102    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340105    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340108    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340111    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340114    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340117    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340120    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340122    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340125    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340128    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340130    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340133    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340136    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.353055 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340138    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340141    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340144    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340147    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340150    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340152    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340155    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340158    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340160    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340163    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340165    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340168    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340171    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340173    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340177    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340180    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340182    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340185    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340187    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.353546 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340191    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340194    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340196    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340199    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340201    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340204    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340206    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340209    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340212    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340215    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340217    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340220    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340223    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340227    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340230    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340234    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340237    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340240    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340243    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.354008 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340245    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340248    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340250    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.340279    4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340411    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340415    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340418    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340421    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340424    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340428    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340430    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340433    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340435    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340438    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340441    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340443    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.354468 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340446    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340449    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340470    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340475    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340478    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340481    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340483    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340486    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340488    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340491    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340494    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340496    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340499    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340501    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340504    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340507    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340510    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340512    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340515    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.355027 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340518    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340521    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340523    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340526    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340529    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340531    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340534    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340537    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340540    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340542    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340545    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340548    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340550    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340553    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340555    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340558    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340560    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340563    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340565    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340568    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.355797 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340570    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340573    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340575    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340578    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340580    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340583    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340585    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340588    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340590    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340593    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340595    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340598    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340601    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340603    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340606    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340608    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340611    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340613    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340616    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340618    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.356387 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340621    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340624    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340626    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340629    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340632    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340634    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340637    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340639    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340642    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340645    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340651    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340655    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340658    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340661    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340664    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340667    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340669    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340672    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340674    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.356981 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340677    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340680    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340682    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340685    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340687    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340690    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340692    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340695    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340699    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340702    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340705    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340708    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340711    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340714    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340718    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.357442 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340721    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.340745    4519 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340777    4519 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340781    4519 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340783    4519 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340786    4519 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340789    4519 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340791    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340799    4519 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340802    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340804    4519 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340807    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340810    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340812    4519 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340815    4519 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340818    4519 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:36.357822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340820    4519 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340823    4519 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340826    4519 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340828    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340831    4519 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340834    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340836    4519 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340839    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340841    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340844    4519 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340847    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340849    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340852    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340854    4519 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340857    4519 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340860    4519 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340863    4519 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340865    4519 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340868    4519 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:36.358259 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340871    4519 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340874    4519 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340876    4519 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340878    4519 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340881    4519 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340886    4519 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340889    4519 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340891    4519 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340894    4519 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340897    4519 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340899    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340902    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340905    4519 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340907    4519 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340910    4519 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340912    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340915    4519 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340917    4519 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340920    4519 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340922    4519 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:36.358749 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340925    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340927    4519 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340930    4519 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340933    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340935    4519 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340938    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340940    4519 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340943    4519 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340945    4519 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340949    4519 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340953    4519 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340955    4519 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340958    4519 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340961    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340963    4519 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340966    4519 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340968    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340972    4519 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340975    4519 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340977    4519 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:36.359220 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340980    4519 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340983    4519 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340986    4519 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340989    4519 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340991    4519 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340994    4519 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340996    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.340999    4519 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341003    4519 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341007    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341010    4519 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341014    4519 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341016    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341019    4519 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341021    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341024    4519 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341026    4519 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341029    4519 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:36.359787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341031    4519 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341034    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341037    4519 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341041    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341043    4519 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341047    4519 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341049    4519 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341053    4519 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341057    4519 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341060    4519 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341063    4519 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341066    4519 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341069    4519 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341072    4519 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:36.341074    4519 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.341136    4519 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342175    4519 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342235    4519 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 16:02:36.360211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342419    4519 server.go:1014] "Starting client certificate rotation"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342578    4519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342629    4519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 15:56:55 +0000 UTC" deadline="2028-05-22 09:41:04.581101818 +0000 UTC"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342664    4519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13649h38m28.238440918s"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.342929    4519 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.343980    4519 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.345449    4519 log.go:25] "Validated CRI v1 runtime API"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.347056    4519 log.go:25] "Validated CRI v1 image API"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.347792    4519 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 16:02:36.360717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.349310    4519 fs.go:135] Filesystem UUIDs: map[113b7fbb-6d32-47d9-bbb4-9a3398174aab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 92ef52f4-ec37-4675-843c-672a3f74d545:/dev/nvme0n1p3]
Oct 31 16:02:36.360954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.349329    4519 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e170cf5-c15f-45bd-9d7d-db33a914f7cc/volumes/kubernetes.io~projected/kube-api-access-trvxr:{mountpoint:/var/lib/kubelet/pods/0e170cf5-c15f-45bd-9d7d-db33a914f7cc/volumes/kubernetes.io~projected/kube-api-access-trvxr major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e170cf5-c15f-45bd-9d7d-db33a914f7cc/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0e170cf5-c15f-45bd-9d7d-db33a914f7cc/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1476215c-1e66-46ba-a910-58fb73446da7/volumes/kubernetes.io~projected/kube-api-access-z2v8v:{mountpoint:/var/lib/kubelet/pods/1476215c-1e66-46ba-a910-58fb73446da7/volumes/kubernetes.io~projected/kube-api-access-z2v8v major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/251fef43-4a22-4508-9e24-166a266d95a0/volumes/kubernetes.io~projected/kube-api-access-svhms:{mountpoint:/var/lib/kubelet/pods/251fef43-4a22-4508-9e24-166a266d95a0/volumes/kubernetes.io~projected/kube-api-access-svhms major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28de26f0-048d-48c5-9df2-d1c4c33f4fb2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/28de26f0-048d-48c5-9df2-d1c4c33f4fb2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb7154b-15e9-473c-8091-fd02a0eb5789/volumes/kubernetes.io~projected/kube-api-access-dlw7q:{mountpoint:/var/lib/kubelet/pods/3cb7154b-15e9-473c-8091-fd02a0eb5789/volumes/kubernetes.io~projected/kube-api-access-dlw7q major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bc628c2-0e1f-45c4-89f9-c7a85ea04468/volumes/kubernetes.io~projected/kube-api-access-8p4pc:{mountpoint:/var/lib/kubelet/pods/5bc628c2-0e1f-45c4-89f9-c7a85ea04468/volumes/kubernetes.io~projected/kube-api-access-8p4pc major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volumes/kubernetes.io~projected/kube-api-access-9xzk2:{mountpoint:/var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volumes/kubernetes.io~projected/kube-api-access-9xzk2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7/volumes/kubernetes.io~projected/kube-api-access-r7zz6:{mountpoint:/var/lib/kubelet/pods/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7/volumes/kubernetes.io~projected/kube-api-access-r7zz6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88955cf1-7c74-4198-a5a6-762fc37473f9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/88955cf1-7c74-4198-a5a6-762fc37473f9/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e659030-d532-4842-8462-7b89281cd2f5/volumes/kubernetes.io~projected/kube-api-access-c2rbk:{mountpoint:/var/lib/kubelet/pods/9e659030-d532-4842-8462-7b89281cd2f5/volumes/kubernetes.io~projected/kube-api-access-c2rbk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb3b8c3-9810-4e2a-8d51-6861faa35baf/volumes/kubernetes.io~projected/kube-api-access-84nx8:{mountpoint:/var/lib/kubelet/pods/9fb3b8c3-9810-4e2a-8d51-6861faa35baf/volumes/kubernetes.io~projected/kube-api-access-84nx8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb3b8c3-9810-4e2a-8d51-6861faa35baf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9fb3b8c3-9810-4e2a-8d51-6861faa35baf/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~projected/kube-api-access-vzvqg:{mountpoint:/var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~projected/kube-api-access-vzvqg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdd2188c-94c4-4700-b0ff-9b7cf279911c/volumes/kubernetes.io~projected/kube-api-access-dg4x8:{mountpoint:/var/lib/kubelet/pods/bdd2188c-94c4-4700-b0ff-9b7cf279911c/volumes/kubernetes.io~projected/kube-api-access-dg4x8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c498d605-032d-4c7b-82ff-1098b1935a88/volumes/kubernetes.io~projected/kube-api-access-tkxlf:{mountpoint:/var/lib/kubelet/pods/c498d605-032d-4c7b-82ff-1098b1935a88/volumes/kubernetes.io~projected/kube-api-access-tkxlf major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7cad6ef-ae99-4477-bd4c-e673dd6791ff/volumes/kubernetes.io~projected/kube-api-access-x5dcn:{mountpoint:/var/lib/kubelet/pods/d7cad6ef-ae99-4477-bd4c-e673dd6791ff/volumes/kubernetes.io~projected/kube-api-access-x5dcn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da23b925-afa4-4246-a9f9-38b52246d95a/volumes/kubernetes.io~projected/kube-api-access-ghqcf:{mountpoint:/var/lib/kubelet/pods/da23b925-afa4-4246-a9f9-38b52246d95a/volumes/kubernetes.io~projected/kube-api-access-ghqcf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f92acc0f-003d-425d-861e-a3a949ea34a5/volumes/kubernetes.io~projected/kube-api-access-x9rjm:{mountpoint:/var/lib/kubelet/pods/f92acc0f-003d-425d-861e-a3a949ea34a5/volumes/kubernetes.io~projected/kube-api-access-x9rjm major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5/volumes/kubernetes.io~projected/kube-api-access-drv8b:{mountpoint:/var/lib/kubelet/pods/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5/volumes/kubernetes.io~projected/kube-api-access-drv8b major:0 minor:216 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bbe10c303c61de933b7d003f7d5931988270d5a9baac69a71d3b8236b118b8a9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/327b50d0eff25ef5816f750796de4877d6c4cabb0214f20c11aadc584ff39439/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/b4aa25b5586aea7472529eb68c4d06bfea857bbae305a1cc55b964b89ef2d9f7/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/55ee6bad68ec6101faa88de93bf9ac7bfa08c78bd08d6495e9684a4b22f998a0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f6ad43f5db35db7eac0a7fb48b2140d2e66dc80a129ce30870079f02a14958b8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8dc5fde43b0c234439c8315cd3b216328f389554d2c8e45437f565678490e882/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9c4704a2852b175315b71682e2c5cf448e359318df996edbfa88352fe9de0d0e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/30923a1fa3b9ad64714a89d5c20d3b422fc7298df051ce5511eca585061fbb51/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ac51b042dc8507fd9c15e808faf049b9327c3fddd16925f43a4b3043e4ea32bb/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a50ba8d9d8785701cc55b4c0fa76d674721d8d3be57d501fc8674b49c33e52c1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f3a927a67b5004b0eee795be076523e0d530e5f5033da3fbcdd5bc5a05f24cce/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b952753394e0e4bc77fff250bbb480b8798081928c94056ee1941961df1f39fe/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/99ce835efed6bff90447d27258eeded78f161f9a0f8e254d21b364512934c56f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6829ab1325431a3e82b3c817ab7c4881000911ceda6ab55b28054b09d198b581/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b3b17f8d26b2b325f04fbf3883f80df8c893602f394ff170a280297a338bec14/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f82f45843ea98e477b364adcc20475e94cfe5ea294d68a96cd5585e018d0a760/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/077aaae26ed82125603ef584ea35d011584e7bf590745267a3666cd75779f62c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9f8eeda752badc87f778aa3629ca72e2d45b547620f268b2e1f7aadbe8c5fc84/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/075118c9d4a17a658521344009ca19e3788df0c5bac6d4c8dd4b6f279fc33c7d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/aef4e0f70d7857cc80a4f189e2d92272cbdf7b04ccec0562419b998164b15cc5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/48adc82100a849aaab8527e7cba39c3554b9c5e7c98bd6fe28844ceccb6e75ff/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/8d632558c8f1582bb6aaca3ae54c0ff1b1b8cfcc64f37272978ddea2321247d0/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/4b08b126b14053ab130ff0ef4ca2374137a6486295a475a9f37ee37bc834a4ce/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/a6768bd70d5eebdc4575a447b2be697add21ae1df308b1a7eae45d1093b6ca4b/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/dfcc6f63791c901c8ffcc4bbe1168c07965793c9c09c994e082f5e47de3232b1/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/51f9082ab5c7b874d8a2156afcbc1e8d4e597a9d0f772d73674204820c61c124/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/13c78837ccce27544a479cc6e98842588a3f4e2c0d858534f95316f0abe31693/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/d1196cd90dbe6a36f928f514de24471326952c4c4f1b8552ff544e7903486038/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/a4952537a4294c78b6fb4dca90892709f3230ee4d0f4bece4b52e32bd523c1a3/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/58a6236a58cfa7b859733bcd114f16bc0e73dd934edf2c77c68fb5e9b3f2496f/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/b7519be2e74f1587b73bbb89cd6bd5811664a508435a394294e2b97bc3925b3d/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/60949248a0ea0fe1d54b1fbe3be3d9042d087097318310c0aa72906e0d11f0dd/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d9c601881068d24801f3ff80e5ef2b39666691ac963fb5b39fcff8eb35389476/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6740cb87e7c12b3d1739eed687d3bd00e353f7a63452c4580fa1225e45c185e4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0230df56c4895293c96ce5530bd92d637814b769619d6badbcb05a2996e5e075/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/cc948859fee2170d095174a6d7cf9c8e3d114ea94d488c29d8e69628bde17d7f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/28305bf049d37ade03c89742281bf43d8c74b11f01c379f5dcb43e6288bd93a5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1e24fe680b447358b7c86ab6590c1196fd452854a11a2fe9b62244ac2a815a78/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/159611e36885cab87086154d4bf71a4042ea565d7e8f13f1c7b747ba7ed82105/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/65abb9bf9b867b4bda551d3c8282b41917915218091ec8ff133843ad0b736f43/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/90ee861547031fad79fb68a548ac3ca71777965a9c6fdbfaf734dfede7cdc1ff/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bb60f5518e83c59b4d6f548f17cd8e778a7dd8ceabf0c0b348e0b32cd0bfb102/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.356885    4519 manager.go:217] Machine: {Timestamp:2026-10-31 16:02:36.35638304 +0000 UTC m=+0.056312344 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099947 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec222fdb132d150bfe59a819f8d49ade SystemUUID:ec222fdb-132d-150b-fe59-a819f8d49ade BootID:624b77bd-eb2c-4453-badb-6239a90b2113 Filesystems:[{Device:/var/lib/kubelet/pods/28de26f0-048d-48c5-9df2-d1c4c33f4fb2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c498d605-032d-4c7b-82ff-1098b1935a88/volumes/kubernetes.io~projected/kube-api-access-tkxlf DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/919f6544498e8ca8a3bc13276861c33e4b22b7a63cf02f8552d0897eceeb2940/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f92acc0f-003d-425d-861e-a3a949ea34a5/volumes/kubernetes.io~projected/kube-api-access-x9rjm DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7cad6ef-ae99-4477-bd4c-e673dd6791ff/volumes/kubernetes.io~projected/kube-api-access-x5dcn DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e170cf5-c15f-45bd-9d7d-db33a914f7cc/volumes/kubernetes.io~projected/kube-api-access-trvxr DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5/volumes/kubernetes.io~projected/kube-api-access-drv8b DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f426c2333d87175e41a6e2859c19dd5923ee53f3a92950dc450a66a6599444ef/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7/volumes/kubernetes.io~projected/kube-api-access-r7zz6 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c3e6876a1f16c26eaaad6e252f14494b897955babca9b4784e83ed71faed306/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb3b8c3-9810-4e2a-8d51-6861faa35baf/volumes/kubernetes.io~projected/kube-api-access-84nx8 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb7154b-15e9-473c-8091-fd02a0eb5789/volumes/kubernetes.io~projected/kube-api-access-dlw7q DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdd2188c-94c4-4700-b0ff-9b7cf279911c/volumes/kubernetes.io~projected/kube-api-access-dg4x8 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88955cf1-7c74-4198-a5a6-762fc37473f9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f12bad6cc2e6bff615e9af906e1ada4cf2e7e075b9921caed2420dc12d46b74/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e026a902b2819e5e58da07aab06bf41468c258ea38753e9bbaa6bb1733ad248/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6b468af14cc4c0ff248ca4e7595c15e98aff014500de2ad88a407b71304a020/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb3b8c3-9810-4e2a-8d51-6861faa35baf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ec24d4718ea00c8bd6441207ab641fbef49fd092813b86d50142ee612ee38d7/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bc628c2-0e1f-45c4-89f9-c7a85ea04468/volumes/kubernetes.io~projected/kube-api-access-8p4pc DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00505bc950afda6b36a4f11e59914957c91d0a50e3611d7326d99053005d504c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1476215c-1e66-46ba-a910-58fb73446da7/volumes/kubernetes.io~projected/kube-api-access-z2v8v DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volumes/kubernetes.io~projected/kube-api-access-9xzk2 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da23b925-afa4-4246-a9f9-38b52246d95a/volumes/kubernetes.io~projected/kube-api-access-ghqcf DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2beb745-0c6b-4060-86fd-0e31575d1342/volumes/kubernetes.io~projected/kube-api-access-vzvqg DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e659030-d532-4842-8462-7b89281cd2f5/volumes/kubernetes.io~projected/kube-api-access-c2rbk DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b3901157e53850e1ca3c78022ee202daa447f889be62b3b3b44262d71ce6797/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68e7747c-9ba3-420e-aba4-25f0308c6a1e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0e170cf5-c15f-45bd-9d7d-db33a914f7cc/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b3790584b7248800f4e61c6f8aa08ffb215b8ddd9f56ff3f22f8646f2111deb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dd70697c87ba0a260bf587f6411f01d0d647e3cea0e599ee95334e34bb282eb/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/251fef43-4a22-4508-9e24-166a266d95a0/volumes/kubernetes.io~projected/kube-api-access-svhms DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a8eb9fb7d8d9257e29674b471529169f84823431ab65de0231ac2da9f82c74d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67cd342158aaf8970f9474c4bcdf987442244adeaedc10586f01fb8849a843ad/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2dd70697c87ba0a MacAddress:6e:ee:9b:35:7e:9e Speed:10000 Mtu:8901} {Name:2ec24d4718ea00c MacAddress:22:53:d4:5a:fc:41 Speed:10000 Mtu:8901} {Name:67cd342158aaf89 MacAddress:e6:47:1b:52:8f:5f Speed:10000 Mtu:8901} {Name:b6b468af14cc4c0 MacAddress:36:3e:48:dc:b2:e6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ce:cf:db:45 Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:b1:4a:36:35:f3 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:ce:cf:db:45 Speed:0 Mtu:9001} {Name:f426c2333d87175 MacAddress:6a:d1:77:06:b5:d1 Speed:10000 Mtu:8901} {Name:f524c7482e199a1 MacAddress:ee:ed:ef:e8:e1:84 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5a:bc:dd:f8:cb:15 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:b5:86:fb:5b:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357142    4519 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357183    4519 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357564    4519 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357586    4519 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-180.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357709    4519 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357716    4519 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357726    4519 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357745    4519 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.357828    4519 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358008    4519 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358111    4519 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358142    4519 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358155    4519 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358165    4519 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358174    4519 kubelet.go:405] "Adding apiserver pod source"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358183    4519 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358881    4519 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.358891    4519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:02:36.361101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.359333    4519 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 16:02:36.390883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.390756    4519 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 16:02:36.390883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.390767    4519 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:02:36.390883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.390856    4519 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 16:02:36.391049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391038    4519 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 16:02:36.391082 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391068    4519 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 16:02:36.391187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391175    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 16:02:36.391235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391197    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 16:02:36.391235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391205    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 16:02:36.391235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391215    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 16:02:36.391235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391229    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391240    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391248    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391253    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391261    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391267    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391275    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391284    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391308    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 16:02:36.391362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.391314    4519 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 16:02:36.394753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.394739    4519 server.go:1263] "Started kubelet"
Oct 31 16:02:36.394812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.394787    4519 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 16:02:36.395216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.394897    4519 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 16:02:36.395216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.395032    4519 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 16:02:36.395322 ip-10-0-129-180 systemd[1]: Started Kubernetes Kubelet.
Oct 31 16:02:36.395514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.395367    4519 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 16:02:36.396484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.396446    4519 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.398131    4519 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.399080    4519 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.399298    4519 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.399539    4519 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:57 +0000 UTC" deadline="2028-06-28 02:41:48.582860163 +0000 UTC"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.399576    4519 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14530h39m12.183287476s"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.399609    4519 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.399656    4519 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.399699    4519 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401768    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e170cf5-c15f-45bd-9d7d-db33a914f7cc" volumeName="kubernetes.io/projected/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-kube-api-access-trvxr" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401811    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e170cf5-c15f-45bd-9d7d-db33a914f7cc" volumeName="kubernetes.io/secret/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-key" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401836    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e7747c-9ba3-420e-aba4-25f0308c6a1e" volumeName="kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-config" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401850    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2beb745-0c6b-4060-86fd-0e31575d1342" volumeName="kubernetes.io/projected/b2beb745-0c6b-4060-86fd-0e31575d1342-kube-api-access-vzvqg" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401871    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb3b8c3-9810-4e2a-8d51-6861faa35baf" volumeName="kubernetes.io/secret/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-cert" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401882    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb7154b-15e9-473c-8091-fd02a0eb5789" volumeName="kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401901    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e7747c-9ba3-420e-aba4-25f0308c6a1e" volumeName="kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401921    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" volumeName="kubernetes.io/projected/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-kube-api-access-r7zz6" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401953    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1476215c-1e66-46ba-a910-58fb73446da7" volumeName="kubernetes.io/projected/1476215c-1e66-46ba-a910-58fb73446da7-kube-api-access-z2v8v" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.401986    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e7747c-9ba3-420e-aba4-25f0308c6a1e" volumeName="kubernetes.io/secret/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402003    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" volumeName="kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-binary-copy" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402064    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2beb745-0c6b-4060-86fd-0e31575d1342" volumeName="kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-tmp" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402080    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7cad6ef-ae99-4477-bd4c-e673dd6791ff" volumeName="kubernetes.io/projected/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-kube-api-access-x5dcn" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402096    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da23b925-afa4-4246-a9f9-38b52246d95a" volumeName="kubernetes.io/empty-dir/da23b925-afa4-4246-a9f9-38b52246d95a-tmp-dir" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402138    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251fef43-4a22-4508-9e24-166a266d95a0" volumeName="kubernetes.io/projected/251fef43-4a22-4508-9e24-166a266d95a0-kube-api-access-svhms" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402153    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" volumeName="kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402169    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88955cf1-7c74-4198-a5a6-762fc37473f9" volumeName="kubernetes.io/configmap/88955cf1-7c74-4198-a5a6-762fc37473f9-konnectivity-ca" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402220    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c498d605-032d-4c7b-82ff-1098b1935a88" volumeName="kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402240    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c498d605-032d-4c7b-82ff-1098b1935a88" volumeName="kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402255    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc628c2-0e1f-45c4-89f9-c7a85ea04468" volumeName="kubernetes.io/projected/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-kube-api-access-8p4pc" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402282    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88955cf1-7c74-4198-a5a6-762fc37473f9" volumeName="kubernetes.io/secret/88955cf1-7c74-4198-a5a6-762fc37473f9-agent-certs" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402299    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb3b8c3-9810-4e2a-8d51-6861faa35baf" volumeName="kubernetes.io/projected/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-kube-api-access-84nx8" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402311    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdd2188c-94c4-4700-b0ff-9b7cf279911c" volumeName="kubernetes.io/configmap/bdd2188c-94c4-4700-b0ff-9b7cf279911c-serviceca" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402337    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdd2188c-94c4-4700-b0ff-9b7cf279911c" volumeName="kubernetes.io/projected/bdd2188c-94c4-4700-b0ff-9b7cf279911c-kube-api-access-dg4x8" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402349    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c498d605-032d-4c7b-82ff-1098b1935a88" volumeName="kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402404    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5" volumeName="kubernetes.io/empty-dir/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-data-volume" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402444    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e170cf5-c15f-45bd-9d7d-db33a914f7cc" volumeName="kubernetes.io/configmap/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-cabundle" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402479    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28de26f0-048d-48c5-9df2-d1c4c33f4fb2" volumeName="kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402520    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc628c2-0e1f-45c4-89f9-c7a85ea04468" volumeName="kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cni-binary-copy" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402567    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e7747c-9ba3-420e-aba4-25f0308c6a1e" volumeName="kubernetes.io/projected/68e7747c-9ba3-420e-aba4-25f0308c6a1e-kube-api-access-9xzk2" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402579    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e659030-d532-4842-8462-7b89281cd2f5" volumeName="kubernetes.io/projected/9e659030-d532-4842-8462-7b89281cd2f5-kube-api-access-c2rbk" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402626    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5" volumeName="kubernetes.io/projected/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-api-access-drv8b" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402654    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1476215c-1e66-46ba-a910-58fb73446da7" volumeName="kubernetes.io/empty-dir/1476215c-1e66-46ba-a910-58fb73446da7-tmp-dir" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402698    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" volumeName="kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402717    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7cad6ef-ae99-4477-bd4c-e673dd6791ff" volumeName="kubernetes.io/configmap/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-iptables-alerter-script" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402740    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da23b925-afa4-4246-a9f9-38b52246d95a" volumeName="kubernetes.io/projected/da23b925-afa4-4246-a9f9-38b52246d95a-kube-api-access-ghqcf" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402753    4519 factory.go:55] Registering systemd factory
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402771    4519 factory.go:223] Registration of the systemd container factory successfully
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402754    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92acc0f-003d-425d-861e-a3a949ea34a5" volumeName="kubernetes.io/projected/f92acc0f-003d-425d-861e-a3a949ea34a5-kube-api-access-x9rjm" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402854    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5" volumeName="kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402888    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1476215c-1e66-46ba-a910-58fb73446da7" volumeName="kubernetes.io/configmap/1476215c-1e66-46ba-a910-58fb73446da7-config-volume" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402907    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc628c2-0e1f-45c4-89f9-c7a85ea04468" volumeName="kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-daemon-config" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402928    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68e7747c-9ba3-420e-aba4-25f0308c6a1e" volumeName="kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-env-overrides" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402942    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2beb745-0c6b-4060-86fd-0e31575d1342" volumeName="kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-tuned" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402959    4519 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c498d605-032d-4c7b-82ff-1098b1935a88" volumeName="kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf" seLinuxMountContext=""
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402975    4519 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.402983    4519 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.403084    4519 factory.go:153] Registering CRI-O factory
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.403103    4519 factory.go:223] Registration of the crio container factory successfully
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.403201    4519 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.403229    4519 factory.go:103] Registering Raw factory
Oct 31 16:02:36.403501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.403272    4519 manager.go:1199] Started watching for new ooms in manager
Oct 31 16:02:36.407240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.403813    4519 manager.go:319] Starting recovery of all containers
Oct 31 16:02:36.429659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.429605    4519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 16:02:36.430862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.430842    4519 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 16:02:36.430862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.430862    4519 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 16:02:36.430862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.430878    4519 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 16:02:36.431074 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.430918    4519 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 16:02:36.441379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.441343    4519 generic.go:356] "Generic (PLEG): container finished" podID="78bd4376b5e50178846f4b8cc0416cb3" containerID="ab45be8c5f59eae8ee3ce1dc720a7bdff4874beb01e0731609e873cb00ee83ed" exitCode=0
Oct 31 16:02:36.449361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.449333    4519 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="779fc7c234848d9a8a7effa96b5b81e00c205d2d2338699537dd26d7f1f365e6" exitCode=0
Oct 31 16:02:36.449361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.449358    4519 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="d6ffdfc95a79a7d1796aaf91ae27b6a05027cbe9a34bb382f312aed2ad718070" exitCode=0
Oct 31 16:02:36.449551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.449368    4519 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="0d623ccd0832a8bfb5be1f8d7680b4ee25e85a3a5e7253a6f1849dafc675aaf4" exitCode=0
Oct 31 16:02:36.449551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.449377    4519 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="22e5fe6d5fa293d1d3a9a3bf623563dcc7146900c35f3658562ae9b2f3ca7ba2" exitCode=0
Oct 31 16:02:36.449551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.449384    4519 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="879bf3c730502b12b4adc7fa2a8e0d4827b38e575f767335e415df83d2e4e703" exitCode=0
Oct 31 16:02:36.449551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.449389    4519 generic.go:356] "Generic (PLEG): container finished" podID="7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7" containerID="0e56a70b3206c203fa512305f14d6016e69abb81dcb4e3d93e44e768b7836ebb" exitCode=0
Oct 31 16:02:36.473726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.473707    4519 manager.go:324] Recovery completed
Oct 31 16:02:36.487226 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.487208    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.488320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.488272    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.488320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.488298    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.488320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.488309    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.489729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489712    4519 cpu_manager.go:226] "Starting" policy="none"
Oct 31 16:02:36.489729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489725    4519 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 16:02:36.489729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489726    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.489899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489741    4519 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 16:02:36.489899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489864    4519 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 16:02:36.489899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489875    4519 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 16:02:36.489899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489894    4519 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 16:02:36.490019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489902    4519 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 16:02:36.490019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489908    4519 policy_none.go:50] "Start"
Oct 31 16:02:36.490019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489916    4519 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 16:02:36.490019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.489925    4519 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 16:02:36.490883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.490870    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.490953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.490898    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.490953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.490913    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.500306 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.500289    4519 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:02:36.506891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.506876    4519 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 16:02:36.506940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.506893    4519 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 16:02:36.506940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.506901    4519 policy_none.go:44] "Start"
Oct 31 16:02:36.510865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.510842    4519 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 16:02:36.513361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513347    4519 manager.go:341] "Starting Device Plugin manager"
Oct 31 16:02:36.513435 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.513373    4519 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 16:02:36.513435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513385    4519 server.go:87] "Starting device plugin registration server"
Oct 31 16:02:36.513730 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513715    4519 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 16:02:36.513792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513730    4519 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 16:02:36.513851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513840    4519 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 16:02:36.513981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513943    4519 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 16:02:36.514048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513986    4519 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 16:02:36.514048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.513993    4519 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 16:02:36.514832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.514743    4519 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T16:02:36.513976455Z","UUID":"deb401ab-77e0-42d9-b0a0-54e42de6fc21","Handler":null,"Name":"","Endpoint":""}
Oct 31 16:02:36.515368 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.515347    4519 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 16:02:36.515470 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.515389    4519 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-180.us-west-2.compute.internal\" not found"
Oct 31 16:02:36.516034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.516018    4519 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 16:02:36.516125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.516042    4519 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 16:02:36.531071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.531030    4519 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"]
Oct 31 16:02:36.531166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.531147    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.532017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.532003    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.532064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.532032    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.532064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.532044    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.532263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.532254    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.532391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.532372    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.533023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533002    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.533142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533030    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.533142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533045    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.533212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533046    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.533212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533188    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.533212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533204    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.533324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"78bd4376b5e50178846f4b8cc0416cb3","Type":"ContainerStarted","Data":"25fb02465d55da832bcacc416d091619f790ca3b466171a3c5f5c13fb724b548"}
Oct 31 16:02:36.533367 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.533335    4519 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.533367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533342    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"78bd4376b5e50178846f4b8cc0416cb3","Type":"ContainerDied","Data":"ab45be8c5f59eae8ee3ce1dc720a7bdff4874beb01e0731609e873cb00ee83ed"}
Oct 31 16:02:36.533367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533360    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"78bd4376b5e50178846f4b8cc0416cb3","Type":"ContainerStarted","Data":"3937e824627cf6f5d2d7f97ad5aee72525e680b34dbb7dd8a7f829ee5fd4dc9d"}
Oct 31 16:02:36.533474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533313    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.533474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533411    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"29bfc3859f40051644e6484149c81648","Type":"ContainerStarted","Data":"6d3842dea956b1395a7d811933b46447f12e305e16bde49c659f1a5ee8e0d8c5"}
Oct 31 16:02:36.533474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.533429    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal" event={"ID":"29bfc3859f40051644e6484149c81648","Type":"ContainerStarted","Data":"219be93492df69b7d377044d51e3cd1dbc8ae79f48e7502f283bb191d52acda4"}
Oct 31 16:02:36.534128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.534114    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.534195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.534136    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.534195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.534161    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.534300 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.534288    4519 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-180.us-west-2.compute.internal\" not found" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.555399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.555382    4519 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 16:02:36.555889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.555874    4519 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:36.555937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.555889    4519 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:36.556018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.556006    4519 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:36.600046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.600021    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.603426 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.603409    4519 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:36.614547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.614527    4519 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:36.615520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.615507    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:36.615579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.615533    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:36.615579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.615543    4519 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-180.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:36.615643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.615581    4519 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.654683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.654660    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:36.654783 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.654708    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.654783 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.654723    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.655803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.655778    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.655906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.655805    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29bfc3859f40051644e6484149c81648-config\") pod \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"29bfc3859f40051644e6484149c81648\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.655906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.655825    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.710174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.710151    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:36.710299 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:36.710189    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.717724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.717707    4519 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.717778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.717769    4519 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.756573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.756515    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.756573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.756543    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.756708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.756574    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29bfc3859f40051644e6484149c81648-config\") pod \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"29bfc3859f40051644e6484149c81648\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.756708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.756594    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.756708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.756612    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29bfc3859f40051644e6484149c81648-config\") pod \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"29bfc3859f40051644e6484149c81648\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:36.756708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:36.756651    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78bd4376b5e50178846f4b8cc0416cb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" (UID: \"78bd4376b5e50178846f4b8cc0416cb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:37.358671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.358646    4519 apiserver.go:52] "Watching apiserver"
Oct 31 16:02:37.456789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.456759    4519 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 16:02:37.457213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.457194    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-78cj7-pgttm","openshift-multus/network-metrics-daemon-h6rw9","openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt","openshift-dns/dns-default-nfm25","openshift-dns/node-resolver-c6ssr","openshift-image-registry/node-ca-cg6j2","openshift-network-diagnostics/network-check-target-cfrr7","kube-system/global-pull-secret-syncer-5n9tx","openshift-ovn-kubernetes/ovnkube-node-j6bw8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal","kube-system/konnectivity-agent-b4fr7","openshift-multus/multus-additional-cni-plugins-qhr6p","openshift-service-ca/service-ca-5fffc97bdf-6wgwt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk","openshift-network-operator/iptables-alerter-2fr6l","openshift-cluster-node-tuning-operator/tuned-vv5k4","openshift-ingress-canary/ingress-canary-bwjhn","openshift-insights/insights-runtime-extractor-jc94s","kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal","openshift-multus/multus-gk2gp"]
Oct 31 16:02:37.457774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.457746    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:37.463836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.463815    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.464984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.464965    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.478955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.478938    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:37.479045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.479037    4519 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:37.501137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.501089    4519 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 16:02:37.509031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509006    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 16:02:37.509130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509034    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.509130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509065    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 16:02:37.509130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509067    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.509289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509135    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 16:02:37.509289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509163    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fd4\""
Oct 31 16:02:37.509289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509227    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 16:02:37.509412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509337    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 16:02:37.509412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509346    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wg2f6\""
Oct 31 16:02:37.509412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.509340    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnhm9\""
Oct 31 16:02:37.510250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.510234    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f9xg5\""
Oct 31 16:02:37.510576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.510563    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.511867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511847    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jqwtc\""
Oct 31 16:02:37.511867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511862    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.512005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511866    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.512005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511849    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tcbkl\""
Oct 31 16:02:37.512005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511849    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.512005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511892    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.512005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511947    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.512005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.511962    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.512248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512048    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msflc\""
Oct 31 16:02:37.512248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512075    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnjtb\""
Oct 31 16:02:37.512248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512117    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.512248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512131    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4fqcg\""
Oct 31 16:02:37.512248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512206    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.512248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512228    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512273    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512286    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512276    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgxcq\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512300    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512343    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512413    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.512494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512448    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 16:02:37.512729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512624    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 16:02:37.512729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512638    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 16:02:37.512729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512661    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d4rvf\""
Oct 31 16:02:37.512838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512746    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.512838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512757    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 16:02:37.512838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512783    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9wrg2\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512868    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512884    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512908    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512910    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tp4vw\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512927    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512933    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 16:02:37.512985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512911    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.513275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.512996    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 16:02:37.513275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.513041    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.513275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.513157    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mwvsn\""
Oct 31 16:02:37.513275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.513168    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.555837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555815    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555856    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555872    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555862    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555860    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555878    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555935    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555899    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.555997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.555968    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qmh6f\""
Oct 31 16:02:37.556341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.556051    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xbg8d\""
Oct 31 16:02:37.556341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.556113    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lhk9l\""
Oct 31 16:02:37.556341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.556152    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 16:02:37.556341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.556321    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 16:02:37.556341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.556330    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7g99\""
Oct 31 16:02:37.556521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.556415    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 16:02:37.559104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.559088    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 16:02:37.561279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561264    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:37.561368 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.561315    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:37.561416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561375    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-modprobe-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.561416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561396    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-lib-modules\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.561527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561417    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-tmp\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.561527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561434    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.561527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561465    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.561668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561573    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-tmp\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.561668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561582    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqcf\" (UniqueName: \"kubernetes.io/projected/da23b925-afa4-4246-a9f9-38b52246d95a-kube-api-access-ghqcf\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.561668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561656    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd2188c-94c4-4700-b0ff-9b7cf279911c-serviceca\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.561777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561727    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.561777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561750    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1476215c-1e66-46ba-a910-58fb73446da7-tmp-dir\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.561874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561787    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-registration-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.561874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561836    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1476215c-1e66-46ba-a910-58fb73446da7-tmp-dir\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.561953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561879    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvxr\" (UniqueName: \"kubernetes.io/projected/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-kube-api-access-trvxr\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:37.561953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561911    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-k8s-cni-cncf-io\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.561953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561940    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da23b925-afa4-4246-a9f9-38b52246d95a-tmp-dir\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561962    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-dbus\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.561980    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-log-socket\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562009    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2v8v\" (UniqueName: \"kubernetes.io/projected/1476215c-1e66-46ba-a910-58fb73446da7-kube-api-access-z2v8v\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562034    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-iptables-alerter-script\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562058    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-system-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562069    4519 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:37.562088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562080    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-slash\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562085    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd2188c-94c4-4700-b0ff-9b7cf279911c-serviceca\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562122    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da23b925-afa4-4246-a9f9-38b52246d95a-tmp-dir\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562141    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562175    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-host-slash\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.562211    4519 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-180.us-west-2.compute.internal"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562336    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-iptables-alerter-script\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.562391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-cabundle\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:37.562650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562412    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.562650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562499    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-var-lib-kubelet\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.562650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562555    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-host\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.562650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562591    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-cabundle\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:37.562650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562596    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/b2beb745-0c6b-4060-86fd-0e31575d1342-kube-api-access-vzvqg\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.562795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562648    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-multus-certs\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.562795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562673    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rbk\" (UniqueName: \"kubernetes.io/projected/9e659030-d532-4842-8462-7b89281cd2f5-kube-api-access-c2rbk\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:37.562795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562692    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-kubelet\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.562795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562727    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.562795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562765    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.562932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562799    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.562932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562826    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-systemd\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.562932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562919    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-run\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.563021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-sys\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.563021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562960    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cni-binary-copy\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.563021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.562958    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.563021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4x8\" (UniqueName: \"kubernetes.io/projected/bdd2188c-94c4-4700-b0ff-9b7cf279911c-kube-api-access-dg4x8\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.563125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563065    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-netns\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563157    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzk2\" (UniqueName: \"kubernetes.io/projected/68e7747c-9ba3-420e-aba4-25f0308c6a1e-kube-api-access-9xzk2\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563177    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cni-binary-copy\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.563206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563186    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-system-cni-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.563303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-conf\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.563341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563328    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-device-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.563374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563362    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-data-volume\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.563434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563423    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-netd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563439    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-data-volume\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.563482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563450    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563498    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-config\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.563598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563554    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.563627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563613    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-conf-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.563676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563660    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-daemon-config\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.563733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563722    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da23b925-afa4-4246-a9f9-38b52246d95a-hosts-file\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.563733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563723    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-config\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.563818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563788    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-systemd-units\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563818    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-var-lib-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563846    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-etc-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.563879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563862    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-daemon-config\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.563879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563874    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zz6\" (UniqueName: \"kubernetes.io/projected/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-kube-api-access-r7zz6\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.563969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563897    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rjm\" (UniqueName: \"kubernetes.io/projected/f92acc0f-003d-425d-861e-a3a949ea34a5-kube-api-access-x9rjm\") pod \"migrator-66df5d496b-6lznt\" (UID: \"f92acc0f-003d-425d-861e-a3a949ea34a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:37.563969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563914    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.563969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563928    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88955cf1-7c74-4198-a5a6-762fc37473f9-agent-certs\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:37.563969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563942    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88955cf1-7c74-4198-a5a6-762fc37473f9-konnectivity-ca\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:37.563969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563965    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-multus\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.563990    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-bin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564079    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-etc-kubernetes\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564101    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.564132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564121    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-bin\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.564239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564143    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-cert\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:37.564239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564172    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.564239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564175    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88955cf1-7c74-4198-a5a6-762fc37473f9-konnectivity-ca\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:37.564239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564196    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.564239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564217    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88955cf1-7c74-4198-a5a6-762fc37473f9-agent-certs\") pod \"konnectivity-agent-b4fr7\" (UID: \"88955cf1-7c74-4198-a5a6-762fc37473f9\") " pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:02:37.564368 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564277    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:37.564368 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564350    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-cert\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:37.564420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564407    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cnibin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564433    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-socket-dir-parent\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564473    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.564525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564482    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-netns\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564514    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-kubelet\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.564603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564534    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.564603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564557    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1476215c-1e66-46ba-a910-58fb73446da7-config-volume\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.564663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564643    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.564691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564657    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.564691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564674    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysconfig\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.564746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564709    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1476215c-1e66-46ba-a910-58fb73446da7-config-volume\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.564746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564714    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.564746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564736    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:37.564825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564767    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd2188c-94c4-4700-b0ff-9b7cf279911c-host\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.564825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564785    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-systemd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.564825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564802    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.564907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564824    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-tuned\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.564907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564847    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-socket-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.564907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564871    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-sys-fs\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.564907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564894    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-hostroot\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564926    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4pc\" (UniqueName: \"kubernetes.io/projected/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-kube-api-access-8p4pc\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564954    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-node-log\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564970    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564977    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-tuned\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.564980    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-script-lib\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565011    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dcn\" (UniqueName: \"kubernetes.io/projected/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-kube-api-access-x5dcn\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.565027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565029    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhms\" (UniqueName: \"kubernetes.io/projected/251fef43-4a22-4508-9e24-166a266d95a0-kube-api-access-svhms\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565139    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-crio-socket\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565158    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drv8b\" (UniqueName: \"kubernetes.io/projected/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-api-access-drv8b\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565172    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-kubelet-config\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565191    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cnibin\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565214    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-kubernetes\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565244    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-ovnkube-script-lib\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565251    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565280    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-key\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565298    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-ovn\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-env-overrides\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565329    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nx8\" (UniqueName: \"kubernetes.io/projected/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-kube-api-access-84nx8\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:37.565624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565355    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.565624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565380    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565407    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.565624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565447    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68e7747c-9ba3-420e-aba4-25f0308c6a1e-env-overrides\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.565624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565488    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-signing-key\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:37.565781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.565674    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-original-pull-secret\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.614989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.614956    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2v8v\" (UniqueName: \"kubernetes.io/projected/1476215c-1e66-46ba-a910-58fb73446da7-kube-api-access-z2v8v\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.615111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.615067    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghqcf\" (UniqueName: \"kubernetes.io/projected/da23b925-afa4-4246-a9f9-38b52246d95a-kube-api-access-ghqcf\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.615218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.615199    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvxr\" (UniqueName: \"kubernetes.io/projected/0e170cf5-c15f-45bd-9d7d-db33a914f7cc-kube-api-access-trvxr\") pod \"service-ca-5fffc97bdf-6wgwt\" (UID: \"0e170cf5-c15f-45bd-9d7d-db33a914f7cc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt"
Oct 31 16:02:37.616413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.616394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/b2beb745-0c6b-4060-86fd-0e31575d1342-kube-api-access-vzvqg\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.616507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.616446    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rbk\" (UniqueName: \"kubernetes.io/projected/9e659030-d532-4842-8462-7b89281cd2f5-kube-api-access-c2rbk\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:37.616550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.616536    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4x8\" (UniqueName: \"kubernetes.io/projected/bdd2188c-94c4-4700-b0ff-9b7cf279911c-kube-api-access-dg4x8\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.616590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.616534    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzk2\" (UniqueName: \"kubernetes.io/projected/68e7747c-9ba3-420e-aba4-25f0308c6a1e-kube-api-access-9xzk2\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.617017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.617002    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rjm\" (UniqueName: \"kubernetes.io/projected/f92acc0f-003d-425d-861e-a3a949ea34a5-kube-api-access-x9rjm\") pod \"migrator-66df5d496b-6lznt\" (UID: \"f92acc0f-003d-425d-861e-a3a949ea34a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lznt"
Oct 31 16:02:37.617423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.617403    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dcn\" (UniqueName: \"kubernetes.io/projected/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-kube-api-access-x5dcn\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.617562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.617542    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4pc\" (UniqueName: \"kubernetes.io/projected/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-kube-api-access-8p4pc\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.617639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.617624    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nx8\" (UniqueName: \"kubernetes.io/projected/9fb3b8c3-9810-4e2a-8d51-6861faa35baf-kube-api-access-84nx8\") pod \"ingress-canary-bwjhn\" (UID: \"9fb3b8c3-9810-4e2a-8d51-6861faa35baf\") " pod="openshift-ingress-canary/ingress-canary-bwjhn"
Oct 31 16:02:37.617729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.617714    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlw7q\" (UniqueName: \"kubernetes.io/projected/3cb7154b-15e9-473c-8091-fd02a0eb5789-kube-api-access-dlw7q\") pod \"network-check-target-cfrr7\" (UID: \"3cb7154b-15e9-473c-8091-fd02a0eb5789\") " pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:37.618227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.618209    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zz6\" (UniqueName: \"kubernetes.io/projected/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-kube-api-access-r7zz6\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.618346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.618324    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drv8b\" (UniqueName: \"kubernetes.io/projected/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-kube-api-access-drv8b\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.618541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.618517    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhms\" (UniqueName: \"kubernetes.io/projected/251fef43-4a22-4508-9e24-166a266d95a0-kube-api-access-svhms\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.618666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.618651    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") pod \"periodic-gathering-78cj7-pgttm\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") " pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:02:37.665660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665636    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-k8s-cni-cncf-io\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.665660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665660    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-dbus\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665675    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-log-socket\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665690    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-system-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665740    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-log-socket\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665746    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-slash\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665738    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-k8s-cni-cncf-io\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665775    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-host-slash\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665779    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-slash\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665749    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-dbus\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.665796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665793    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-var-lib-kubelet\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665809    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7cad6ef-ae99-4477-bd4c-e673dd6791ff-host-slash\") pod \"iptables-alerter-2fr6l\" (UID: \"d7cad6ef-ae99-4477-bd4c-e673dd6791ff\") " pod="openshift-network-operator/iptables-alerter-2fr6l"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665819    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-host\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665824    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-var-lib-kubelet\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665781    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-system-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665835    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-multus-certs\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665847    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-host\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665850    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-kubelet\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665864    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-multus-certs\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665898    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-kubelet\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665929    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665954    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-systemd\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665929    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665976    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-run\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-sys\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666016    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-systemd\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666019    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-run\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.665977    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666029    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-netns\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666055    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-system-cni-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666065    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-sys\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666074    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-run-netns\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666079    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-conf\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666103    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-device-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666108    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-system-cni-dir\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666128    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-netd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-device-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666178    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-conf\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666178    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-netd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666187    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666211    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-conf-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666223    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-cni-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666227    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da23b925-afa4-4246-a9f9-38b52246d95a-hosts-file\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666251    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-conf-dir\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666256    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-systemd-units\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666257    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666273    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-var-lib-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666280    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da23b925-afa4-4246-a9f9-38b52246d95a-hosts-file\") pod \"node-resolver-c6ssr\" (UID: \"da23b925-afa4-4246-a9f9-38b52246d95a\") " pod="openshift-dns/node-resolver-c6ssr"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666287    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-systemd-units\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666291    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-etc-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666314    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-multus\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666315    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-var-lib-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666328    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-bin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666341    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-etc-kubernetes\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666343    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-etc-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666358    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-bin\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666374    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-bin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666377    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666387    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-cni-multus\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666396    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cnibin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666389    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-etc-kubernetes\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666419    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-socket-dir-parent\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666425    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-cnibin\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666433    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-host-cni-bin\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666439    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-netns\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666470    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-multus-socket-dir-parent\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666475    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-run-netns\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666490    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-kubelet\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666517    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysconfig\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666527    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysctl-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666564    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-host-var-lib-kubelet\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666564    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd2188c-94c4-4700-b0ff-9b7cf279911c-host\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666598    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd2188c-94c4-4700-b0ff-9b7cf279911c-host\") pod \"node-ca-cg6j2\" (UID: \"bdd2188c-94c4-4700-b0ff-9b7cf279911c\") " pod="openshift-image-registry/node-ca-cg6j2"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666579    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-sysconfig\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666621    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-systemd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.666655    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666672    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-systemd\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666700    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-socket-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666654    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-socket-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.666743    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.16672685 +0000 UTC m=+1.866656162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : secret "metrics-daemon-secret" not found
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666762    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-sys-fs\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666778    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-hostroot\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666793    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-node-log\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666815    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-crio-socket\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666830    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-kubelet-config\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666843    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cnibin\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666858    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-kubernetes\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666870    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bc628c2-0e1f-45c4-89f9-c7a85ea04468-hostroot\") pod \"multus-gk2gp\" (UID: \"5bc628c2-0e1f-45c4-89f9-c7a85ea04468\") " pod="openshift-multus/multus-gk2gp"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666884    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.666860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666886    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-sys-fs\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666917    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-node-log\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666897    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-crio-socket\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666921    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7-cnibin\") pod \"multus-additional-cni-plugins-qhr6p\" (UID: \"7e69cb7a-20ad-47dd-8a50-c5a6a3299fd7\") " pod="openshift-multus/multus-additional-cni-plugins-qhr6p"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666936    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28de26f0-048d-48c5-9df2-d1c4c33f4fb2-kubelet-config\") pod \"global-pull-secret-syncer-5n9tx\" (UID: \"28de26f0-048d-48c5-9df2-d1c4c33f4fb2\") " pod="kube-system/global-pull-secret-syncer-5n9tx"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666949    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-kubernetes\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666974    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-ovn\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.666999    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-ovn\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667027    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68e7747c-9ba3-420e-aba4-25f0308c6a1e-run-openvswitch\") pod \"ovnkube-node-j6bw8\" (UID: \"68e7747c-9ba3-420e-aba4-25f0308c6a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667042    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-modprobe-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667060    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-lib-modules\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667073    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667088    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667111    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667132    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-registration-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667148    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-etc-modprobe-d\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667151    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.667183    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667193    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/251fef43-4a22-4508-9e24-166a266d95a0-registration-dir\") pod \"aws-ebs-csi-driver-node-dbnpk\" (UID: \"251fef43-4a22-4508-9e24-166a266d95a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbnpk"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.667199    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.667193    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2beb745-0c6b-4060-86fd-0e31575d1342-lib-modules\") pod \"tuned-vv5k4\" (UID: \"b2beb745-0c6b-4060-86fd-0e31575d1342\") " pod="openshift-cluster-node-tuning-operator/tuned-vv5k4"
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.667234    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.16721826 +0000 UTC m=+1.867147618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:37.668584 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:37.667248    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.16724141 +0000 UTC m=+1.867170698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:37.767002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:37.766985    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:02:38.170539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:38.170440    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:38.170539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:38.170502    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:38.170539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:38.170523    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:38.170792 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:38.170618    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:38.170792 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:38.170632    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:38.170792 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:38.170633    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:38.170792 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:38.170676    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:39.170665051 +0000 UTC m=+2.870594352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:38.170792 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:38.170688    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:39.170682539 +0000 UTC m=+2.870611828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : secret "metrics-daemon-secret" not found
Oct 31 16:02:38.170792 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:38.170708    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:39.170692181 +0000 UTC m=+2.870621485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:38.351994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:38.351969    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:39.039468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:39.039430    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c6ssr_da23b925-afa4-4246-a9f9-38b52246d95a/dns-node-resolver/0.log"
Oct 31 16:02:39.178034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:39.177954    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:39.178034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:39.178005    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:39.178201 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:39.178118    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:39.178201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:39.178138    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:39.178201 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:39.178118    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:39.178201 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:39.178187    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.178167638 +0000 UTC m=+4.878096941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:39.178339 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:39.178249    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.178224459 +0000 UTC m=+4.878153762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:39.178339 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:39.178282    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:39.178339 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:39.178328    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.178316964 +0000 UTC m=+4.878246266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : secret "metrics-daemon-secret" not found
Oct 31 16:02:39.485334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:39.485295    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6wgwt" event={"ID":"0e170cf5-c15f-45bd-9d7d-db33a914f7cc","Type":"ContainerStarted","Data":"933ac26deeaa851416dfbfe0992b8cdae2ebe483021235e3d26f9dccdd22c622"}
Oct 31 16:02:39.839539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:39.839469    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cg6j2_bdd2188c-94c4-4700-b0ff-9b7cf279911c/node-ca/0.log"
Oct 31 16:02:40.839509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:40.839479    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bwjhn_9fb3b8c3-9810-4e2a-8d51-6861faa35baf/serve-healthcheck-canary/0.log"
Oct 31 16:02:41.192687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:41.192601    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:41.192687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:41.192651    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:41.192687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:41.192674    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:41.192963 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:41.192767    4519 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:41.192963 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:41.192837    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls podName:fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:45.19282319 +0000 UTC m=+8.892752491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jc94s" (UID: "fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:41.192963 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:41.192767    4519 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:41.192963 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:41.192769    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:41.193122 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:41.192935    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls podName:1476215c-1e66-46ba-a910-58fb73446da7 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:45.192914522 +0000 UTC m=+8.892843828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls") pod "dns-default-nfm25" (UID: "1476215c-1e66-46ba-a910-58fb73446da7") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:41.193122 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:41.193039    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:45.193025659 +0000 UTC m=+8.892954949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : secret "metrics-daemon-secret" not found
Oct 31 16:02:41.440054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:41.440018    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6lznt_f92acc0f-003d-425d-861e-a3a949ea34a5/migrator/0.log"
Oct 31 16:02:45.220297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.220262    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:45.220806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.220305    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:45.220806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.220324    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:45.220806 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:45.220421    4519 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:45.220806 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:45.220522    4519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs podName:9e659030-d532-4842-8462-7b89281cd2f5 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:53.220502533 +0000 UTC m=+16.920431839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs") pod "network-metrics-daemon-h6rw9" (UID: "9e659030-d532-4842-8462-7b89281cd2f5") : secret "metrics-daemon-secret" not found
Oct 31 16:02:45.220806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.220684    4519 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 16:02:45.222775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.222754    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1476215c-1e66-46ba-a910-58fb73446da7-metrics-tls\") pod \"dns-default-nfm25\" (UID: \"1476215c-1e66-46ba-a910-58fb73446da7\") " pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:45.222775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.222763    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jc94s\" (UID: \"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5\") " pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:45.264994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.264954    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:45.266086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.266033    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jc94s"
Oct 31 16:02:45.720184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.720162    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nfm25"]
Oct 31 16:02:45.759179 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:45.759155    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2f76dc_92ec_4ba9_9ea7_36de06fa19c5.slice/crio-6cddb7268624dd67075acfb9c63dbf64be74890f5680e8238ef6b2784acc053f WatchSource:0}: Error finding container 6cddb7268624dd67075acfb9c63dbf64be74890f5680e8238ef6b2784acc053f: Status 404 returned error can't find the container with id 6cddb7268624dd67075acfb9c63dbf64be74890f5680e8238ef6b2784acc053f
Oct 31 16:02:45.767359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:45.767338    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jc94s"]
Oct 31 16:02:46.506438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:46.506403    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfm25" event={"ID":"1476215c-1e66-46ba-a910-58fb73446da7","Type":"ContainerStarted","Data":"993131d393732f2a8ff1bdcb05107b977a126b18328b9d063f9b67991c0095fe"}
Oct 31 16:02:46.507866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:46.507841    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc94s" event={"ID":"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5","Type":"ContainerStarted","Data":"a3fd5eb89195c44af0c0508f214c23309401822029f67cd2971e8829f096a69d"}
Oct 31 16:02:46.507980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:46.507886    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc94s" event={"ID":"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5","Type":"ContainerStarted","Data":"6cddb7268624dd67075acfb9c63dbf64be74890f5680e8238ef6b2784acc053f"}
Oct 31 16:02:46.858912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:46.858824    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:02:47.513801 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:47.513761    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc94s" event={"ID":"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5","Type":"ContainerStarted","Data":"ae7e7f74e956cbe8142e61abefd254cd5520fff6e6ad642a9dbb6c77522265b8"}
Oct 31 16:02:47.589629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:47.589602    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:02:47.589860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:47.589846    4519 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 16:02:48.519506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:48.519447    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfm25" event={"ID":"1476215c-1e66-46ba-a910-58fb73446da7","Type":"ContainerStarted","Data":"70614b1d77eaa3cb8a2ab897c729cfcf7b8073d00463eedcb35f7ab604971d73"}
Oct 31 16:02:48.519918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:48.519531    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nfm25" event={"ID":"1476215c-1e66-46ba-a910-58fb73446da7","Type":"ContainerStarted","Data":"dcf8c1483db2c3892c92c53b7e676c54ddda1deff56affbec96e9c9125e092ea"}
Oct 31 16:02:48.519918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:48.519601    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:49.524484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:49.524434    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jc94s" event={"ID":"fe2f76dc-92ec-4ba9-9ea7-36de06fa19c5","Type":"ContainerStarted","Data":"00161b2635fc91413fd993c72b378af130c0873627441c98a02a5ccd46ab1c8d"}
Oct 31 16:02:53.278851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:53.278800    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:53.281105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:53.281082    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e659030-d532-4842-8462-7b89281cd2f5-metrics-certs\") pod \"network-metrics-daemon-h6rw9\" (UID: \"9e659030-d532-4842-8462-7b89281cd2f5\") " pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:53.358223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:53.358192    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6rw9"
Oct 31 16:02:53.714936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:53.714908    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6rw9"]
Oct 31 16:02:53.718351 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:53.718325    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e659030_d532_4842_8462_7b89281cd2f5.slice/crio-27df9e266fbf714ea37f5b51e70d6456f4b4ca0a84c32fa11081abbaaa0ec87b WatchSource:0}: Error finding container 27df9e266fbf714ea37f5b51e70d6456f4b4ca0a84c32fa11081abbaaa0ec87b: Status 404 returned error can't find the container with id 27df9e266fbf714ea37f5b51e70d6456f4b4ca0a84c32fa11081abbaaa0ec87b
Oct 31 16:02:54.002298 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:02:54.002207    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:02:54.543420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:54.543384    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6rw9" event={"ID":"9e659030-d532-4842-8462-7b89281cd2f5","Type":"ContainerStarted","Data":"27df9e266fbf714ea37f5b51e70d6456f4b4ca0a84c32fa11081abbaaa0ec87b"}
Oct 31 16:02:55.548485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:55.548385    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6rw9" event={"ID":"9e659030-d532-4842-8462-7b89281cd2f5","Type":"ContainerStarted","Data":"52ed196314b0deaa91b4233cf9dd3ea36f4abd55f8d15665303ec48d3c0b8eb5"}
Oct 31 16:02:55.548485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:55.548440    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6rw9" event={"ID":"9e659030-d532-4842-8462-7b89281cd2f5","Type":"ContainerStarted","Data":"bf08788aae62a18135f20d3f89c796212ce3ba08d1755f3c78c0819be7db1a78"}
Oct 31 16:02:58.527311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:58.527279    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nfm25"
Oct 31 16:02:59.010213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.010184    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"]
Oct 31 16:02:59.013354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.013340    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.065942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.065904    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 16:02:59.112514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.112488    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 16:02:59.112514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.112486    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 16:02:59.112949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.112922    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 16:02:59.112949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.112949    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rf2dj\""
Oct 31 16:02:59.113099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.113001    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 16:02:59.113555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.113538    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"]
Oct 31 16:02:59.117306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.117291    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/825b5af5-4574-43af-85d3-43ff9677289f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.117360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.117321    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/825b5af5-4574-43af-85d3-43ff9677289f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.117360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.117338    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/825b5af5-4574-43af-85d3-43ff9677289f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.117360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.117354    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx57d\" (UniqueName: \"kubernetes.io/projected/825b5af5-4574-43af-85d3-43ff9677289f-kube-api-access-cx57d\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.218620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.218580    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/825b5af5-4574-43af-85d3-43ff9677289f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.218620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.218621    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/825b5af5-4574-43af-85d3-43ff9677289f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.218804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.218640    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx57d\" (UniqueName: \"kubernetes.io/projected/825b5af5-4574-43af-85d3-43ff9677289f-kube-api-access-cx57d\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.218804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.218756    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/825b5af5-4574-43af-85d3-43ff9677289f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.219258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.219237    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/825b5af5-4574-43af-85d3-43ff9677289f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.221040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.221018    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/825b5af5-4574-43af-85d3-43ff9677289f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.221040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.221027    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/825b5af5-4574-43af-85d3-43ff9677289f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.272159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.272108    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx57d\" (UniqueName: \"kubernetes.io/projected/825b5af5-4574-43af-85d3-43ff9677289f-kube-api-access-cx57d\") pod \"prometheus-operator-9cd89b74c-cpxsw\" (UID: \"825b5af5-4574-43af-85d3-43ff9677289f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.326040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.326011    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"
Oct 31 16:02:59.682830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:02:59.682806    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw"]
Oct 31 16:02:59.685620 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:02:59.685590    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825b5af5_4574_43af_85d3_43ff9677289f.slice/crio-a70b3f72e286d0e1033fd462e38ab3710fe561c5fcf83f36f770bcf5b7056076 WatchSource:0}: Error finding container a70b3f72e286d0e1033fd462e38ab3710fe561c5fcf83f36f770bcf5b7056076: Status 404 returned error can't find the container with id a70b3f72e286d0e1033fd462e38ab3710fe561c5fcf83f36f770bcf5b7056076
Oct 31 16:03:00.564223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:00.564191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw" event={"ID":"825b5af5-4574-43af-85d3-43ff9677289f","Type":"ContainerStarted","Data":"a70b3f72e286d0e1033fd462e38ab3710fe561c5fcf83f36f770bcf5b7056076"}
Oct 31 16:03:02.573042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:02.573002    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw" event={"ID":"825b5af5-4574-43af-85d3-43ff9677289f","Type":"ContainerStarted","Data":"08bc48a21247bfe7bc6724e77e968fe7f258eb37e04837c6e5d6b1c3dd46a8d6"}
Oct 31 16:03:02.573531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:02.573054    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw" event={"ID":"825b5af5-4574-43af-85d3-43ff9677289f","Type":"ContainerStarted","Data":"bedb581630ced70a21f19ce319a140efa7d204f96b92e696de98a18889b0e4ec"}
Oct 31 16:03:02.681284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:02.681233    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-cpxsw" podStartSLOduration=2.7818023739999997 podStartE2EDuration="4.681219763s" totalImagesPullingTime="1.899417389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:02.680230855 +0000 UTC m=+26.380160166" watchObservedRunningTime="2026-10-31 16:03:02.681219763 +0000 UTC m=+26.381149073"
Oct 31 16:03:04.592664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:04.592638    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b4fr7"
Oct 31 16:03:04.753713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:04.753685    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6bw8"
Oct 31 16:03:05.402224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.402190    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"]
Oct 31 16:03:05.435418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.435393    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.462778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.462743    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e69d240c-0010-405d-994a-6c819de2a2c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.462955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.462801    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e69d240c-0010-405d-994a-6c819de2a2c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.462955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.462833    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhxzs\" (UniqueName: \"kubernetes.io/projected/e69d240c-0010-405d-994a-6c819de2a2c7-kube-api-access-mhxzs\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.462955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.462872    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69d240c-0010-405d-994a-6c819de2a2c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.463810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.463784    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"]
Oct 31 16:03:05.490929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.490901    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 16:03:05.491082 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.491029    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 16:03:05.491134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.491104    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t5dfl\""
Oct 31 16:03:05.500931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.500904    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kt6cf"]
Oct 31 16:03:05.518074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.518052    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563470    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-sys\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563527    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e69d240c-0010-405d-994a-6c819de2a2c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.563622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563554    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-wtmp\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563582    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-textfile\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563616    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e69d240c-0010-405d-994a-6c819de2a2c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.563717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563704    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhxzs\" (UniqueName: \"kubernetes.io/projected/e69d240c-0010-405d-994a-6c819de2a2c7-kube-api-access-mhxzs\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.563802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563776    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/605636d0-4624-47b5-b74a-8a4b09f582f5-metrics-client-ca\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563853    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-tls\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.563918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.563893    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69d240c-0010-405d-994a-6c819de2a2c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.564079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.564061    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bg4c\" (UniqueName: \"kubernetes.io/projected/605636d0-4624-47b5-b74a-8a4b09f582f5-kube-api-access-9bg4c\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.564177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.564093    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.564177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.564122    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-root\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.564271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.564209    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e69d240c-0010-405d-994a-6c819de2a2c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.566168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.566138    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e69d240c-0010-405d-994a-6c819de2a2c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.566260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.566148    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69d240c-0010-405d-994a-6c819de2a2c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.583964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.583941    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 16:03:05.591349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.591332    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 16:03:05.591508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.591491    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 16:03:05.591552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.591527    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jlhfg\""
Oct 31 16:03:05.617009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.616987    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhxzs\" (UniqueName: \"kubernetes.io/projected/e69d240c-0010-405d-994a-6c819de2a2c7-kube-api-access-mhxzs\") pod \"openshift-state-metrics-7bd7489957-hkh4x\" (UID: \"e69d240c-0010-405d-994a-6c819de2a2c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.664421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664354    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664396    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-root\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664489    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-root\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664530    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-sys\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664582    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-wtmp\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664612    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-textfile\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664618    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-sys\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664677    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/605636d0-4624-47b5-b74a-8a4b09f582f5-metrics-client-ca\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664740    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-tls\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.664821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664778    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bg4c\" (UniqueName: \"kubernetes.io/projected/605636d0-4624-47b5-b74a-8a4b09f582f5-kube-api-access-9bg4c\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.665058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664817    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-wtmp\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.665058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.664904    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-textfile\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.665058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.665050    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.665509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.665328    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/605636d0-4624-47b5-b74a-8a4b09f582f5-metrics-client-ca\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.667251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.667227    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.667535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.667513    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/605636d0-4624-47b5-b74a-8a4b09f582f5-node-exporter-tls\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.718301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.718276    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bg4c\" (UniqueName: \"kubernetes.io/projected/605636d0-4624-47b5-b74a-8a4b09f582f5-kube-api-access-9bg4c\") pod \"node-exporter-kt6cf\" (UID: \"605636d0-4624-47b5-b74a-8a4b09f582f5\") " pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.785421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.785393    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"
Oct 31 16:03:05.860658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:05.860436    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kt6cf"
Oct 31 16:03:05.885263 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:05.885237    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605636d0_4624_47b5_b74a_8a4b09f582f5.slice/crio-872335e47dc4f75660ad46b634252ef2ec1b118906cdea8ec2334c8f31320191 WatchSource:0}: Error finding container 872335e47dc4f75660ad46b634252ef2ec1b118906cdea8ec2334c8f31320191: Status 404 returned error can't find the container with id 872335e47dc4f75660ad46b634252ef2ec1b118906cdea8ec2334c8f31320191
Oct 31 16:03:06.205249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:06.205225    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x"]
Oct 31 16:03:06.207378 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:06.207349    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69d240c_0010_405d_994a_6c819de2a2c7.slice/crio-e211ea6baac0efbda4fbf2fe2ae6cb962cac2184829a2a9478506cc7914e63b7 WatchSource:0}: Error finding container e211ea6baac0efbda4fbf2fe2ae6cb962cac2184829a2a9478506cc7914e63b7: Status 404 returned error can't find the container with id e211ea6baac0efbda4fbf2fe2ae6cb962cac2184829a2a9478506cc7914e63b7
Oct 31 16:03:06.589962 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:06.589921    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x" event={"ID":"e69d240c-0010-405d-994a-6c819de2a2c7","Type":"ContainerStarted","Data":"82ad91b458d0c14350983beb0ac5e85cf4245ce921fcae03e8633113df7dc552"}
Oct 31 16:03:06.590155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:06.590016    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x" event={"ID":"e69d240c-0010-405d-994a-6c819de2a2c7","Type":"ContainerStarted","Data":"a42a9de2f5f8a5adae9b9c280c24e3c4ad477733b7475557caad7b2d56ecb4fc"}
Oct 31 16:03:06.590155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:06.590040    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x" event={"ID":"e69d240c-0010-405d-994a-6c819de2a2c7","Type":"ContainerStarted","Data":"e211ea6baac0efbda4fbf2fe2ae6cb962cac2184829a2a9478506cc7914e63b7"}
Oct 31 16:03:06.591126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:06.591101    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kt6cf" event={"ID":"605636d0-4624-47b5-b74a-8a4b09f582f5","Type":"ContainerStarted","Data":"872335e47dc4f75660ad46b634252ef2ec1b118906cdea8ec2334c8f31320191"}
Oct 31 16:03:06.862602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:06.862528    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cfrr7"
Oct 31 16:03:07.982818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:07.982685    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-867775f966-8rs4k"]
Oct 31 16:03:08.014798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.014772    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.042438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.042410    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867775f966-8rs4k"]
Oct 31 16:03:08.083925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.083889    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65c6561-1883-49ed-9cd6-d29f927b4f0f-tmp\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.083928    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-tls\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.083999    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.084060    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.084090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf685\" (UniqueName: \"kubernetes.io/projected/d65c6561-1883-49ed-9cd6-d29f927b4f0f-kube-api-access-nf685\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.084109    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-grpc-tls\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.084212    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d65c6561-1883-49ed-9cd6-d29f927b4f0f-metrics-client-ca\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.084250    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.084355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.084311    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.088201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088170    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 16:03:08.088201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088174    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bfqt3q79pan52\""
Oct 31 16:03:08.088660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088616    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 16:03:08.088660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088618    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 16:03:08.088660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088645    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 16:03:08.088857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088669    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sf7lf\""
Oct 31 16:03:08.088857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.088697    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 16:03:08.185070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185038    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185078    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185108    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65c6561-1883-49ed-9cd6-d29f927b4f0f-tmp\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185142    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-tls\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185180    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185213    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185237    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf685\" (UniqueName: \"kubernetes.io/projected/d65c6561-1883-49ed-9cd6-d29f927b4f0f-kube-api-access-nf685\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185260    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-grpc-tls\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185317    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d65c6561-1883-49ed-9cd6-d29f927b4f0f-metrics-client-ca\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.185778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.185754    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65c6561-1883-49ed-9cd6-d29f927b4f0f-tmp\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.186276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.186246    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d65c6561-1883-49ed-9cd6-d29f927b4f0f-metrics-client-ca\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.187560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.187525    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.188330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.188302    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.188497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.188308    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-tls\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.188497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.188413    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.188626 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.188567    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-grpc-tls\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.189043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.189015    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d65c6561-1883-49ed-9cd6-d29f927b4f0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.239185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.239126    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf685\" (UniqueName: \"kubernetes.io/projected/d65c6561-1883-49ed-9cd6-d29f927b4f0f-kube-api-access-nf685\") pod \"thanos-querier-867775f966-8rs4k\" (UID: \"d65c6561-1883-49ed-9cd6-d29f927b4f0f\") " pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.417183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.417149    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:08.600194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.600156    4519 generic.go:356] "Generic (PLEG): container finished" podID="605636d0-4624-47b5-b74a-8a4b09f582f5" containerID="c2a215eaa9a2b6d75cd6e5a16104413afe0d2f50ddb41d330ede844b842b0811" exitCode=0
Oct 31 16:03:08.600194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.600196    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kt6cf" event={"ID":"605636d0-4624-47b5-b74a-8a4b09f582f5","Type":"ContainerDied","Data":"c2a215eaa9a2b6d75cd6e5a16104413afe0d2f50ddb41d330ede844b842b0811"}
Oct 31 16:03:08.890285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:08.890203    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867775f966-8rs4k"]
Oct 31 16:03:08.892757 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:08.892733    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65c6561_1883_49ed_9cd6_d29f927b4f0f.slice/crio-380d67655fedcb37ce1e9d4090e02e65fbd914a0eaecf0a11dee27c43b9e240c WatchSource:0}: Error finding container 380d67655fedcb37ce1e9d4090e02e65fbd914a0eaecf0a11dee27c43b9e240c: Status 404 returned error can't find the container with id 380d67655fedcb37ce1e9d4090e02e65fbd914a0eaecf0a11dee27c43b9e240c
Oct 31 16:03:09.606436 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:09.606398    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kt6cf" event={"ID":"605636d0-4624-47b5-b74a-8a4b09f582f5","Type":"ContainerStarted","Data":"a323c5ff090c33ed6fb9814ab4348d842f3d2576c428538f0737162bdf89b69d"}
Oct 31 16:03:09.606862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:09.606476    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kt6cf" event={"ID":"605636d0-4624-47b5-b74a-8a4b09f582f5","Type":"ContainerStarted","Data":"6f18c82e814d59284f01939db175cb4eed7119dab8cc51d202e54c0a1fd4b4f5"}
Oct 31 16:03:09.608129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:09.608095    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"380d67655fedcb37ce1e9d4090e02e65fbd914a0eaecf0a11dee27c43b9e240c"}
Oct 31 16:03:09.610160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:09.610132    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x" event={"ID":"e69d240c-0010-405d-994a-6c819de2a2c7","Type":"ContainerStarted","Data":"3e5e674b297e72309f8a33382fa755c1ba738e6b309e6a2f90937a6a7965b5fb"}
Oct 31 16:03:09.715498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:09.715430    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kt6cf" podStartSLOduration=2.547993683 podStartE2EDuration="4.715415604s" totalImagesPullingTime="2.077612128s" totalInitContainerRuntime="89.809793ms" isStatefulPod=true podCreationTimestamp="2026-10-31 16:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:09.714917199 +0000 UTC m=+33.414846510" watchObservedRunningTime="2026-10-31 16:03:09.715415604 +0000 UTC m=+33.415344916"
Oct 31 16:03:09.823480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:09.823414    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hkh4x" podStartSLOduration=2.343877256 podStartE2EDuration="4.823399542s" totalImagesPullingTime="2.479522286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:09.823003077 +0000 UTC m=+33.522932387" watchObservedRunningTime="2026-10-31 16:03:09.823399542 +0000 UTC m=+33.523328853"
Oct 31 16:03:10.180638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.180604    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-s762m"]
Oct 31 16:03:10.202616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.202579    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.236297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.236268    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-s762m"]
Oct 31 16:03:10.248819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.248792    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66558c4d44-cv8cg"]
Oct 31 16:03:10.253859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.253835    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 16:03:10.253859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.253835    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gsh6r\""
Oct 31 16:03:10.266101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.266082    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.301646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301619    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efa8ea09-5a8d-4522-85eb-ae2922cedbb5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-s762m\" (UID: \"efa8ea09-5a8d-4522-85eb-ae2922cedbb5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.301646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301649    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-secret-metrics-server-client-certs\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.301839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301669    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftpl\" (UniqueName: \"kubernetes.io/projected/efa8ea09-5a8d-4522-85eb-ae2922cedbb5-kube-api-access-tftpl\") pod \"monitoring-plugin-77f4f76594-s762m\" (UID: \"efa8ea09-5a8d-4522-85eb-ae2922cedbb5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.301839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301696    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-secret-metrics-server-tls\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.301935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301834    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-client-ca-bundle\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.301935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301867    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fn56\" (UniqueName: \"kubernetes.io/projected/aed3c8c6-be70-4af8-9349-e041cd22d75d-kube-api-access-5fn56\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.301935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.301929    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aed3c8c6-be70-4af8-9349-e041cd22d75d-metrics-server-audit-profiles\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.302090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.302018    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed3c8c6-be70-4af8-9349-e041cd22d75d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.302090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.302071    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aed3c8c6-be70-4af8-9349-e041cd22d75d-audit-log\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.314084 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.314045    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66558c4d44-cv8cg"]
Oct 31 16:03:10.319320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.319296    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 16:03:10.321423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.321387    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 16:03:10.354838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.354585    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cg2xv\""
Oct 31 16:03:10.354838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.354686    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 16:03:10.354838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.354767    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 16:03:10.354838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.354814    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fbp892godpl6s\""
Oct 31 16:03:10.402614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402578    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed3c8c6-be70-4af8-9349-e041cd22d75d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.402614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aed3c8c6-be70-4af8-9349-e041cd22d75d-audit-log\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.402854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402646    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efa8ea09-5a8d-4522-85eb-ae2922cedbb5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-s762m\" (UID: \"efa8ea09-5a8d-4522-85eb-ae2922cedbb5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.402854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402664    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-secret-metrics-server-client-certs\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.402854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402728    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftpl\" (UniqueName: \"kubernetes.io/projected/efa8ea09-5a8d-4522-85eb-ae2922cedbb5-kube-api-access-tftpl\") pod \"monitoring-plugin-77f4f76594-s762m\" (UID: \"efa8ea09-5a8d-4522-85eb-ae2922cedbb5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.402854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402773    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-secret-metrics-server-tls\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.402985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.402963    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-client-ca-bundle\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.403020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.403004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fn56\" (UniqueName: \"kubernetes.io/projected/aed3c8c6-be70-4af8-9349-e041cd22d75d-kube-api-access-5fn56\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.403083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.403062    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aed3c8c6-be70-4af8-9349-e041cd22d75d-audit-log\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.403140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.403065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aed3c8c6-be70-4af8-9349-e041cd22d75d-metrics-server-audit-profiles\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.403691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.403665    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aed3c8c6-be70-4af8-9349-e041cd22d75d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.403966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.403943    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aed3c8c6-be70-4af8-9349-e041cd22d75d-metrics-server-audit-profiles\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.405895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.405869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/efa8ea09-5a8d-4522-85eb-ae2922cedbb5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-s762m\" (UID: \"efa8ea09-5a8d-4522-85eb-ae2922cedbb5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.406019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.405869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-secret-metrics-server-client-certs\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.406019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.405880    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-client-ca-bundle\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.406019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.405997    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aed3c8c6-be70-4af8-9349-e041cd22d75d-secret-metrics-server-tls\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.458770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.457724    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fn56\" (UniqueName: \"kubernetes.io/projected/aed3c8c6-be70-4af8-9349-e041cd22d75d-kube-api-access-5fn56\") pod \"metrics-server-66558c4d44-cv8cg\" (UID: \"aed3c8c6-be70-4af8-9349-e041cd22d75d\") " pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.460296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.460266    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftpl\" (UniqueName: \"kubernetes.io/projected/efa8ea09-5a8d-4522-85eb-ae2922cedbb5-kube-api-access-tftpl\") pod \"monitoring-plugin-77f4f76594-s762m\" (UID: \"efa8ea09-5a8d-4522-85eb-ae2922cedbb5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.530469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.530426    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:10.592947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.592920    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:10.947332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.947296    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-s762m"]
Oct 31 16:03:10.988713 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:10.988685    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa8ea09_5a8d_4522_85eb_ae2922cedbb5.slice/crio-9d14b0b53cb54f20e58b9ad176ee13413b63b22188425fa3b7a2bd4cf4c4032f WatchSource:0}: Error finding container 9d14b0b53cb54f20e58b9ad176ee13413b63b22188425fa3b7a2bd4cf4c4032f: Status 404 returned error can't find the container with id 9d14b0b53cb54f20e58b9ad176ee13413b63b22188425fa3b7a2bd4cf4c4032f
Oct 31 16:03:10.998240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:10.998216    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66558c4d44-cv8cg"]
Oct 31 16:03:11.000444 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:11.000419    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed3c8c6_be70_4af8_9349_e041cd22d75d.slice/crio-7f785669041017f00469b14185480cb5820f2ea614f97f692c6f701b92ab7d0a WatchSource:0}: Error finding container 7f785669041017f00469b14185480cb5820f2ea614f97f692c6f701b92ab7d0a: Status 404 returned error can't find the container with id 7f785669041017f00469b14185480cb5820f2ea614f97f692c6f701b92ab7d0a
Oct 31 16:03:11.131071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.131045    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"]
Oct 31 16:03:11.143975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.143941    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.193284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.193223    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 16:03:11.193284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.193241    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 16:03:11.193556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.193307    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 16:03:11.193556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.193444    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 16:03:11.193697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.193678    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p2hxk\""
Oct 31 16:03:11.211251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211227    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-metrics-client-ca\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211264    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211297    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7zv\" (UniqueName: \"kubernetes.io/projected/a18cb14a-22cf-4097-b27e-28cdb522de11-kube-api-access-sk7zv\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211402    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-serving-certs-ca-bundle\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211432    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-telemeter-client-tls\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211477    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-federate-client-tls\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211510    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-secret-telemeter-client\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.211689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.211540    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.240846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.240669    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 16:03:11.288184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.288149    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 16:03:11.289526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.289500    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"]
Oct 31 16:03:11.312642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312610    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.312807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312661    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-metrics-client-ca\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.312807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312698    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.312807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312731    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7zv\" (UniqueName: \"kubernetes.io/projected/a18cb14a-22cf-4097-b27e-28cdb522de11-kube-api-access-sk7zv\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.312807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312765    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-serving-certs-ca-bundle\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.312807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312786    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-telemeter-client-tls\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.313074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312817    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-federate-client-tls\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.313074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.312865    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-secret-telemeter-client\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.313967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.313941    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-serving-certs-ca-bundle\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.314102 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.314079    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.314297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.314277    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a18cb14a-22cf-4097-b27e-28cdb522de11-metrics-client-ca\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.315871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.315848    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-telemeter-client-tls\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.315980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.315959    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.316030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.315965    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-secret-telemeter-client\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.316475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.316442    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a18cb14a-22cf-4097-b27e-28cdb522de11-federate-client-tls\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.367441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.367409    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7zv\" (UniqueName: \"kubernetes.io/projected/a18cb14a-22cf-4097-b27e-28cdb522de11-kube-api-access-sk7zv\") pod \"telemeter-client-677b7f95bc-zl2nn\" (UID: \"a18cb14a-22cf-4097-b27e-28cdb522de11\") " pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.491574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.491501    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"
Oct 31 16:03:11.619739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.619696    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg" event={"ID":"aed3c8c6-be70-4af8-9349-e041cd22d75d","Type":"ContainerStarted","Data":"7f785669041017f00469b14185480cb5820f2ea614f97f692c6f701b92ab7d0a"}
Oct 31 16:03:11.621178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.621142    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m" event={"ID":"efa8ea09-5a8d-4522-85eb-ae2922cedbb5","Type":"ContainerStarted","Data":"9d14b0b53cb54f20e58b9ad176ee13413b63b22188425fa3b7a2bd4cf4c4032f"}
Oct 31 16:03:11.986090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:11.985781    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-677b7f95bc-zl2nn"]
Oct 31 16:03:11.988675 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:11.988652    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18cb14a_22cf_4097_b27e_28cdb522de11.slice/crio-8f0889a665ba4cc05336698b1aa68bfdd24556946cac690a3dd5fe8a1ea2e0da WatchSource:0}: Error finding container 8f0889a665ba4cc05336698b1aa68bfdd24556946cac690a3dd5fe8a1ea2e0da: Status 404 returned error can't find the container with id 8f0889a665ba4cc05336698b1aa68bfdd24556946cac690a3dd5fe8a1ea2e0da
Oct 31 16:03:12.628289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:12.628203    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"56b03d4138f32fe8aeded8eae1526ac540a7099440b4265d8d8aca31dde08738"}
Oct 31 16:03:12.628478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:12.628337    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"9ace25af18a3f77010e18eda2e12ec6b44fba28ce62a3bee073c734ca23c9878"}
Oct 31 16:03:12.628478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:12.628379    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"3644580e481714a75245ecb39eb9753ec0e058e474eea6488030a73f9a33e8f9"}
Oct 31 16:03:12.629664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:12.629594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn" event={"ID":"a18cb14a-22cf-4097-b27e-28cdb522de11","Type":"ContainerStarted","Data":"8f0889a665ba4cc05336698b1aa68bfdd24556946cac690a3dd5fe8a1ea2e0da"}
Oct 31 16:03:13.634949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:13.634862    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m" event={"ID":"efa8ea09-5a8d-4522-85eb-ae2922cedbb5","Type":"ContainerStarted","Data":"2ba8524cb3928dbfb38fc6ff205c2f7747f64d56dc29850d235308aac48345a5"}
Oct 31 16:03:13.635403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:13.635013    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:13.636685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:13.636651    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg" event={"ID":"aed3c8c6-be70-4af8-9349-e041cd22d75d","Type":"ContainerStarted","Data":"33a513c58c3dd67b822db30437726447731d651b1394a3c9a7bd6c69c087f4b6"}
Oct 31 16:03:13.640941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:13.640917    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m"
Oct 31 16:03:13.741617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:13.741568    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-s762m" podStartSLOduration=1.388030334 podStartE2EDuration="3.741551953s" totalImagesPullingTime="2.353521619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:13.741224373 +0000 UTC m=+37.441153684" watchObservedRunningTime="2026-10-31 16:03:13.741551953 +0000 UTC m=+37.441481263"
Oct 31 16:03:13.956066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:13.955966    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg" podStartSLOduration=1.719051723 podStartE2EDuration="3.955952018s" totalImagesPullingTime="2.236900295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:13.955469141 +0000 UTC m=+37.655398482" watchObservedRunningTime="2026-10-31 16:03:13.955952018 +0000 UTC m=+37.655881329"
Oct 31 16:03:14.289413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.289374    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:03:14.293960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.293932    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.335829    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.335881    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.335907    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbvg8\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.335942    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.335968    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336030    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336053    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336080    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336148    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336183    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336226    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336260    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336284    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336314    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336343    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336369    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336403    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.336813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.336487    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.348771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348724    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 16:03:14.348771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348762    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 16:03:14.348971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348848    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 16:03:14.348971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348873    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 16:03:14.348971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348926    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 16:03:14.349064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348986    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 16:03:14.349064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.348988    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-anhmq90ejg0q1\""
Oct 31 16:03:14.349064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.349003    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 16:03:14.349170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.349152    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9dd78\""
Oct 31 16:03:14.356085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.356066    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 16:03:14.356384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.356329    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 16:03:14.356384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.356355    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 16:03:14.356384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.356369    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 16:03:14.356384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.356379    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 16:03:14.436944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.436912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.436958    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.436992    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437013    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437031    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437049    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437068    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437121    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437158    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437179    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437193    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvg8\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437215    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437229    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437255    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437269    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437284    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437302    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437333    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.437387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.437360    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.439494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.438537    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.439494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.439398    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.441418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.440360    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.441418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.440401    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.441418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.440727    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.441418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.440837    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.441418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.441356    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.442111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.442042    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.442249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.442227    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.443067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.443045    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.443753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.443727    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.443838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.443795    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.444127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.444100    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.444255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.444228    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 16:03:14.444389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.444365    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 16:03:14.444667 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.444645    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.445013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.444991    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.445087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.445033    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.446525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.446505    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:03:14.447294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.447274    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.450803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.450785    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.452642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.452622    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.493415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.493356    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvg8\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8\") pod \"prometheus-k8s-0\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.629083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.629030    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:14.647558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.647489    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"d185ad9701533ef2b3e1a179cc8c1b25c95c970f44ac58ed38e23d25419c8016"}
Oct 31 16:03:14.647961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.647573    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"d0fa1e55db4829d400b4736df8f4231670075e01f5458575e9c4d36df7819cce"}
Oct 31 16:03:14.647961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.647605    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" event={"ID":"d65c6561-1883-49ed-9cd6-d29f927b4f0f","Type":"ContainerStarted","Data":"9f9f1babab94fa63f9a1b36835e634058239f9f9087196d7031aa03c608be3e9"}
Oct 31 16:03:14.761434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:14.761345    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k" podStartSLOduration=2.884706242 podStartE2EDuration="7.761332905s" totalImagesPullingTime="4.876626663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:14.760582102 +0000 UTC m=+38.460511412" watchObservedRunningTime="2026-10-31 16:03:14.761332905 +0000 UTC m=+38.461262216"
Oct 31 16:03:15.055878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:15.055856    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:03:15.057364 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:03:15.057333    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded787f52_41ed_4f94_a4da_984ecfedbc53.slice/crio-28461174d1ef9148bfda7660e956e7325cd827d9b27b832de95eed67ca6d9a4b WatchSource:0}: Error finding container 28461174d1ef9148bfda7660e956e7325cd827d9b27b832de95eed67ca6d9a4b: Status 404 returned error can't find the container with id 28461174d1ef9148bfda7660e956e7325cd827d9b27b832de95eed67ca6d9a4b
Oct 31 16:03:15.652084 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:15.652051    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn" event={"ID":"a18cb14a-22cf-4097-b27e-28cdb522de11","Type":"ContainerStarted","Data":"7a2be3eff449177bdb615d4390cad5c69c7cb246a32c4c89af8d2fd01b880330"}
Oct 31 16:03:15.657123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:15.657086    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"28461174d1ef9148bfda7660e956e7325cd827d9b27b832de95eed67ca6d9a4b"}
Oct 31 16:03:15.657329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:15.657313    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:17.664864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:17.664789    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9" exitCode=0
Oct 31 16:03:17.665250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:17.664876    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"}
Oct 31 16:03:17.667247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:17.667226    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn" event={"ID":"a18cb14a-22cf-4097-b27e-28cdb522de11","Type":"ContainerStarted","Data":"1bb6112c40b0e4bd91ef9c1c47bf2029ce9b62b525b8fc0cfa9301fec86379bb"}
Oct 31 16:03:17.667319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:17.667283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn" event={"ID":"a18cb14a-22cf-4097-b27e-28cdb522de11","Type":"ContainerStarted","Data":"d192bd172de112459c911ed39fc23116c153351acffd0782df08dea30f7f358e"}
Oct 31 16:03:17.911725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:17.911669    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-677b7f95bc-zl2nn" podStartSLOduration=2.324534431 podStartE2EDuration="6.911655054s" totalImagesPullingTime="4.587120623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:17.910491531 +0000 UTC m=+41.610420842" watchObservedRunningTime="2026-10-31 16:03:17.911655054 +0000 UTC m=+41.611584359"
Oct 31 16:03:21.669107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:21.669082    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-867775f966-8rs4k"
Oct 31 16:03:21.685986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:21.685959    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"}
Oct 31 16:03:21.686111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:21.686037    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"}
Oct 31 16:03:21.686111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:21.686083    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"}
Oct 31 16:03:21.686225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:21.686129    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"}
Oct 31 16:03:22.692565 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:22.692533    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"}
Oct 31 16:03:22.692978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:22.692595    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerStarted","Data":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"}
Oct 31 16:03:22.839296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:22.839240    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.476350594 podStartE2EDuration="8.839225906s" totalImagesPullingTime="5.349731105s" totalInitContainerRuntime="13.144207ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:22.80864865 +0000 UTC m=+46.508577960" watchObservedRunningTime="2026-10-31 16:03:22.839225906 +0000 UTC m=+46.539155217"
Oct 31 16:03:24.005721 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:03:24.005489    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:03:24.629563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:24.629527    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:30.593498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:30.593465    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:30.593944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:30.593527    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:31.521251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:31.521208    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:31.541154 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:31.541125    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:31.739916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:31.739889    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:03:50.598635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:50.598606    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:50.602322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:50.602304    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66558c4d44-cv8cg"
Oct 31 16:03:54.003301 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:03:54.003261    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:03:59.829605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:59.829570    4519 generic.go:356] "Generic (PLEG): container finished" podID="c498d605-032d-4c7b-82ff-1098b1935a88" containerID="67b11516633f65ce011f9d5476f7f51282a717ac4bd54815aa3601dec867f59c" exitCode=0
Oct 31 16:03:59.830020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:03:59.829650    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-78cj7-pgttm" event={"ID":"c498d605-032d-4c7b-82ff-1098b1935a88","Type":"ContainerDied","Data":"67b11516633f65ce011f9d5476f7f51282a717ac4bd54815aa3601dec867f59c"}
Oct 31 16:04:00.966393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:00.966372    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:04:01.048945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.048917    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") pod \"c498d605-032d-4c7b-82ff-1098b1935a88\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") "
Oct 31 16:04:01.049097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.048961    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") pod \"c498d605-032d-4c7b-82ff-1098b1935a88\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") "
Oct 31 16:04:01.049097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.049023    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") pod \"c498d605-032d-4c7b-82ff-1098b1935a88\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") "
Oct 31 16:04:01.049201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.049180    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") pod \"c498d605-032d-4c7b-82ff-1098b1935a88\" (UID: \"c498d605-032d-4c7b-82ff-1098b1935a88\") "
Oct 31 16:04:01.049389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.049295    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path" pod "c498d605-032d-4c7b-82ff-1098b1935a88" (UID: "c498d605-032d-4c7b-82ff-1098b1935a88"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:01.049490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.049447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle" pod "c498d605-032d-4c7b-82ff-1098b1935a88" (UID: "c498d605-032d-4c7b-82ff-1098b1935a88"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:01.049532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.049497    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle" pod "c498d605-032d-4c7b-82ff-1098b1935a88" (UID: "c498d605-032d-4c7b-82ff-1098b1935a88"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:01.051214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.051194    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf" pod "c498d605-032d-4c7b-82ff-1098b1935a88" (UID: "c498d605-032d-4c7b-82ff-1098b1935a88"). InnerVolumeSpecName "kube-api-access-tkxlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:01.149929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.149854    4519 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c498d605-032d-4c7b-82ff-1098b1935a88-archives-path\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:01.149929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.149879    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxlf\" (UniqueName: \"kubernetes.io/projected/c498d605-032d-4c7b-82ff-1098b1935a88-kube-api-access-tkxlf\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:01.149929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.149890    4519 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-service-ca-bundle\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:01.149929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.149899    4519 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c498d605-032d-4c7b-82ff-1098b1935a88-trusted-ca-bundle\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:01.840973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.840943    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-78cj7-pgttm" event={"ID":"c498d605-032d-4c7b-82ff-1098b1935a88","Type":"ContainerDied","Data":"f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248"}
Oct 31 16:04:01.840973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.840962    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-78cj7-pgttm"
Oct 31 16:04:01.841178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:01.840988    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f524c7482e199a183411e53f46688f3c05a4fcfd85f96904c3ef501714c36248"
Oct 31 16:04:05.010750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.010717    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:04:05.011217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.011167    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="prometheus" containerID="cri-o://c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc" gracePeriod=600
Oct 31 16:04:05.011296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.011207    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy" containerID="cri-o://fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31" gracePeriod=600
Oct 31 16:04:05.011349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.011212    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="thanos-sidecar" containerID="cri-o://e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a" gracePeriod=600
Oct 31 16:04:05.011349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.011270    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-thanos" containerID="cri-o://29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3" gracePeriod=600
Oct 31 16:04:05.011445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.011264    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-web" containerID="cri-o://e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071" gracePeriod=600
Oct 31 16:04:05.011445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.011277    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="config-reloader" containerID="cri-o://1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599" gracePeriod=600
Oct 31 16:04:05.273234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.273209    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:05.285035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285016    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285049    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285073    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285106    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285136    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285168    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285195    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285237    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285264    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285296    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285334    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285363    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285411    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285446    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285498    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285538    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285577    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.285636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285611    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.286005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285648    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.286005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285686    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca\") pod \"ed787f52-41ed-4f94-a4da-984ecfedbc53\" (UID: \"ed787f52-41ed-4f94-a4da-984ecfedbc53\") "
Oct 31 16:04:05.286005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285816    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.286005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.285958    4519 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.286188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.286159    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.286619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.286395    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.286619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.286471    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:05.286779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.286654    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.287742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.287713    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.287879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.287853    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.288243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.288223    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.288401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.288346    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.288653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.288627    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.288738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.288675    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.288850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.288828    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.289190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.289145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:05.290305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.290273    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.290378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.290305    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.290757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.290739    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:05.291066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.291049    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "kube-api-access-bbvg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:05.291379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.291351    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:05.291586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.291402    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.300792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.300771    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config" pod "ed787f52-41ed-4f94-a4da-984ecfedbc53" (UID: "ed787f52-41ed-4f94-a4da-984ecfedbc53"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:05.387069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387039    4519 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-metrics-client-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387063    4519 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387073    4519 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387083    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbvg8\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-kube-api-access-bbvg8\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387094    4519 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-kube-rbac-proxy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387103    4519 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-web-config\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387112    4519 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387122    4519 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-metrics-client-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387131    4519 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-db\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387139    4519 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387147    4519 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387156    4519 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-config\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387165    4519 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387175    4519 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387184    4519 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed787f52-41ed-4f94-a4da-984ecfedbc53-tls-assets\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387192    4519 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed787f52-41ed-4f94-a4da-984ecfedbc53-config-out\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387199    4519 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-secret-grpc-tls\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387208    4519 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed787f52-41ed-4f94-a4da-984ecfedbc53-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.387283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.387217    4519 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed787f52-41ed-4f94-a4da-984ecfedbc53-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:05.858642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858608    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3" exitCode=0
Oct 31 16:04:05.858642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858638    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31" exitCode=0
Oct 31 16:04:05.858642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858647    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071" exitCode=0
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858656    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a" exitCode=0
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858665    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599" exitCode=0
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858673    4519 generic.go:356] "Generic (PLEG): container finished" podID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc" exitCode=0
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858687    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"}
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"}
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858701    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858790    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"}
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858818    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"}
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"}
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858869    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"}
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858871    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.858899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.858893    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed787f52-41ed-4f94-a4da-984ecfedbc53","Type":"ContainerDied","Data":"28461174d1ef9148bfda7660e956e7325cd827d9b27b832de95eed67ca6d9a4b"}
Oct 31 16:04:05.867575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.867555    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.874486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.874468    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.880766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.880748    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.887086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.887069    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.893205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.893184    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.899841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.899827    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.907880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.907797    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.908162 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.908120    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.908241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.908160    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"} err="failed to get container status \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist"
Oct 31 16:04:05.908241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.908202    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.908844 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.908563    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.908844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.908590    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"} err="failed to get container status \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist"
Oct 31 16:04:05.908844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.908609    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.910567 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.910540    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.910637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.910572    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"} err="failed to get container status \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist"
Oct 31 16:04:05.910637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.910590    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.910882 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.910865    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.910928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.910887    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"} err="failed to get container status \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist"
Oct 31 16:04:05.910928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.910902    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.911130 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.911114    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.911168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911135    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"} err="failed to get container status \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist"
Oct 31 16:04:05.911168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911148    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.911351 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.911336    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.911403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911355    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"} err="failed to get container status \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist"
Oct 31 16:04:05.911403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911375    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.911613 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:05.911598    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.911662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911617    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"} err="failed to get container status \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist"
Oct 31 16:04:05.911662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911629    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.911809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911789    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"} err="failed to get container status \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist"
Oct 31 16:04:05.911857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911809    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.911977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911961    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"} err="failed to get container status \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist"
Oct 31 16:04:05.912020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.911977    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.912170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912155    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"} err="failed to get container status \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist"
Oct 31 16:04:05.912210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912169    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.912359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912342    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"} err="failed to get container status \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist"
Oct 31 16:04:05.912413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912359    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.912555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912541    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"} err="failed to get container status \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist"
Oct 31 16:04:05.912555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912555    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.912727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912710    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"} err="failed to get container status \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist"
Oct 31 16:04:05.912769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912727    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.912976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912961    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"} err="failed to get container status \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist"
Oct 31 16:04:05.913027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.912976    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.913172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913152    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"} err="failed to get container status \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist"
Oct 31 16:04:05.913221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913172    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.913401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913382    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"} err="failed to get container status \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist"
Oct 31 16:04:05.913401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913401    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.913658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913632    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"} err="failed to get container status \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist"
Oct 31 16:04:05.913713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913659    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.913926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913905    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"} err="failed to get container status \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist"
Oct 31 16:04:05.913967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.913929    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.914153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914130    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"} err="failed to get container status \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist"
Oct 31 16:04:05.914194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914153    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.914377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914357    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"} err="failed to get container status \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist"
Oct 31 16:04:05.914482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914378    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.914613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914591    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"} err="failed to get container status \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist"
Oct 31 16:04:05.914681 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914616    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.914914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914885    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"} err="failed to get container status \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist"
Oct 31 16:04:05.914914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.914913    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.915296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.915276    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"} err="failed to get container status \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist"
Oct 31 16:04:05.915375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.915296    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.915607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.915577    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"} err="failed to get container status \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist"
Oct 31 16:04:05.915607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.915600    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.915867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.915845    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"} err="failed to get container status \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist"
Oct 31 16:04:05.915867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.915867    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.916129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.916108    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"} err="failed to get container status \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist"
Oct 31 16:04:05.916129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.916129    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.916411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.916378    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"} err="failed to get container status \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist"
Oct 31 16:04:05.916411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.916404    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.916684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.916661    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"} err="failed to get container status \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist"
Oct 31 16:04:05.916758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.916684    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.917086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917059    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"} err="failed to get container status \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist"
Oct 31 16:04:05.917086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917086    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.917359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917340    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"} err="failed to get container status \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist"
Oct 31 16:04:05.917417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917360    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.917641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917623    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"} err="failed to get container status \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist"
Oct 31 16:04:05.917721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917642    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.917941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917921    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"} err="failed to get container status \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist"
Oct 31 16:04:05.917990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.917941    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.918184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918144    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"} err="failed to get container status \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist"
Oct 31 16:04:05.918259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918185    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.918449 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918425    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"} err="failed to get container status \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist"
Oct 31 16:04:05.918523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918464    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.918700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918681    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"} err="failed to get container status \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist"
Oct 31 16:04:05.918700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918699    4519 scope.go:122] "RemoveContainer" containerID="29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"
Oct 31 16:04:05.918914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918899    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3"} err="failed to get container status \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": rpc error: code = NotFound desc = could not find container \"29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3\": container with ID starting with 29295f114c81901ba394b5ceafc7a02d72c2fb8dea3c03f1f97898e0858e09c3 not found: ID does not exist"
Oct 31 16:04:05.918914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.918913    4519 scope.go:122] "RemoveContainer" containerID="fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"
Oct 31 16:04:05.919096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919074    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31"} err="failed to get container status \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": rpc error: code = NotFound desc = could not find container \"fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31\": container with ID starting with fbec56b411c037da53125007765a7f936c72ced1ad4e6e28ddf54fafc6b62d31 not found: ID does not exist"
Oct 31 16:04:05.919096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919088    4519 scope.go:122] "RemoveContainer" containerID="e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"
Oct 31 16:04:05.919312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919292    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071"} err="failed to get container status \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": rpc error: code = NotFound desc = could not find container \"e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071\": container with ID starting with e6504d12fdc4da3c8558b364f5042537a4bdb0cdcdff14656925aab5b58dc071 not found: ID does not exist"
Oct 31 16:04:05.919379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919313    4519 scope.go:122] "RemoveContainer" containerID="e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"
Oct 31 16:04:05.919553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919536    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a"} err="failed to get container status \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": rpc error: code = NotFound desc = could not find container \"e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a\": container with ID starting with e0d2f14bb5a63bdb26cd85efdbf95b307e3115e696d48aa500b847c7ab638c7a not found: ID does not exist"
Oct 31 16:04:05.919628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919555    4519 scope.go:122] "RemoveContainer" containerID="1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"
Oct 31 16:04:05.919796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919778    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599"} err="failed to get container status \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": rpc error: code = NotFound desc = could not find container \"1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599\": container with ID starting with 1d2e4c6d9ca07178af28c78a0c49d44e770035279f576a69ec380fdf8ae69599 not found: ID does not exist"
Oct 31 16:04:05.919844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919797    4519 scope.go:122] "RemoveContainer" containerID="c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"
Oct 31 16:04:05.920016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.919997    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc"} err="failed to get container status \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": rpc error: code = NotFound desc = could not find container \"c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc\": container with ID starting with c8c804caf389dbb371a7ee1ebac7bdfa376c52f57e2e6a935d7fd84fe5c701fc not found: ID does not exist"
Oct 31 16:04:05.920087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.920016    4519 scope.go:122] "RemoveContainer" containerID="4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"
Oct 31 16:04:05.920216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:05.920199    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9"} err="failed to get container status \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": rpc error: code = NotFound desc = could not find container \"4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9\": container with ID starting with 4d758ba1e6b9a8560363f6294e64abd9cfd7d6727e28065617c788c8a3d758e9 not found: ID does not exist"
Oct 31 16:04:06.018161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.018123    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:04:06.065286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.065258    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:04:06.066036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066016    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:04:06.066301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066289    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="prometheus"
Oct 31 16:04:06.066339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066302    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="prometheus"
Oct 31 16:04:06.066339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066319    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="config-reloader"
Oct 31 16:04:06.066339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066324    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="config-reloader"
Oct 31 16:04:06.066339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066331    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy"
Oct 31 16:04:06.066339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066336    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066343    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-thanos"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066348    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-thanos"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066356    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-web"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066360    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-web"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066373    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c498d605-032d-4c7b-82ff-1098b1935a88" containerName="insights-gathering"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066377    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c498d605-032d-4c7b-82ff-1098b1935a88" containerName="insights-gathering"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066388    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="init-config-reloader"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066393    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="init-config-reloader"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066399    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="thanos-sidecar"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066404    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="thanos-sidecar"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066442    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c498d605-032d-4c7b-82ff-1098b1935a88" containerName="insights-gathering"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066450    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="thanos-sidecar"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066469    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="prometheus"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066476    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="config-reloader"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066481    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066485    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-thanos"
Oct 31 16:04:06.066484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.066492    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" containerName="kube-rbac-proxy-web"
Oct 31 16:04:06.071419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.071404    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091237    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091273    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091307    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091425    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091447    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091473    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091549    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091635    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091676    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-config\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091712    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b09ab819-dfb8-46c1-a23e-53dcd927f337-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091754    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091771    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-web-config\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091789    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091805    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b09ab819-dfb8-46c1-a23e-53dcd927f337-config-out\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091842    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.091900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091890    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.092030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091929    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.092030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwjs\" (UniqueName: \"kubernetes.io/projected/b09ab819-dfb8-46c1-a23e-53dcd927f337-kube-api-access-kjwjs\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.092030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091981    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.092030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.091998    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.121991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.121916    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 16:04:06.121991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.121978    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 16:04:06.122208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122085    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 16:04:06.122208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122098    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 16:04:06.122208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122104    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 16:04:06.122208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122087    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 16:04:06.122208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122156    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 16:04:06.122208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122182    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 16:04:06.122434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122239    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9dd78\""
Oct 31 16:04:06.122434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122270    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 16:04:06.122434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.122353    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-anhmq90ejg0q1\""
Oct 31 16:04:06.152161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.152130    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 16:04:06.152161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.152160    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 16:04:06.152337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.152179    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 16:04:06.192971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.192943    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.192981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.192999    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193016    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193041    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193150    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193195    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-config\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193234    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b09ab819-dfb8-46c1-a23e-53dcd927f337-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193264    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-web-config\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193309    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193337    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b09ab819-dfb8-46c1-a23e-53dcd927f337-config-out\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193359    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193386    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193418    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193444    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwjs\" (UniqueName: \"kubernetes.io/projected/b09ab819-dfb8-46c1-a23e-53dcd927f337-kube-api-access-kjwjs\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193512    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193536    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193551    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.193611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193586    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.194113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.193669    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.194257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.194236    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.194428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.194405    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.194717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.194693    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.194812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.194792    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.195705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.195571    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.196156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.196110    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.196768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.196750    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.197105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.197076    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-web-config\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.197439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.197418    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.197664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.197645    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.197834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.197819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.197951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.197932    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b09ab819-dfb8-46c1-a23e-53dcd927f337-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.198214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.198191    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b09ab819-dfb8-46c1-a23e-53dcd927f337-config-out\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.198397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.198379    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-config\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.198521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.198506    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.199115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.198976    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b09ab819-dfb8-46c1-a23e-53dcd927f337-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.204251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.202358    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 16:04:06.204683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.204652    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 16:04:06.205171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.205152    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:04:06.208647 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.208629    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.214157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.214132    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b09ab819-dfb8-46c1-a23e-53dcd927f337-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.248581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.248551    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwjs\" (UniqueName: \"kubernetes.io/projected/b09ab819-dfb8-46c1-a23e-53dcd927f337-kube-api-access-kjwjs\") pod \"prometheus-k8s-0\" (UID: \"b09ab819-dfb8-46c1-a23e-53dcd927f337\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.395199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.395126    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:06.436372 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.436335    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed787f52-41ed-4f94-a4da-984ecfedbc53" path="/var/lib/kubelet/pods/ed787f52-41ed-4f94-a4da-984ecfedbc53/volumes"
Oct 31 16:04:06.856157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.856133    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 16:04:06.857587 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:04:06.857560    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09ab819_dfb8_46c1_a23e_53dcd927f337.slice/crio-e4922f4402feb918e2c5589c1d805c43267c16a50c9d6db6bee3b27e3959628d WatchSource:0}: Error finding container e4922f4402feb918e2c5589c1d805c43267c16a50c9d6db6bee3b27e3959628d: Status 404 returned error can't find the container with id e4922f4402feb918e2c5589c1d805c43267c16a50c9d6db6bee3b27e3959628d
Oct 31 16:04:06.862897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:06.862862    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"e4922f4402feb918e2c5589c1d805c43267c16a50c9d6db6bee3b27e3959628d"}
Oct 31 16:04:07.868957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:07.868925    4519 generic.go:356] "Generic (PLEG): container finished" podID="b09ab819-dfb8-46c1-a23e-53dcd927f337" containerID="7ae861bae9a9fb36b084e1a2889218c5e2beaeb9fde7c882f9f62339b2175877" exitCode=0
Oct 31 16:04:07.869365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:07.868991    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerDied","Data":"7ae861bae9a9fb36b084e1a2889218c5e2beaeb9fde7c882f9f62339b2175877"}
Oct 31 16:04:08.876132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.876096    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"8b830cf07f4b517643d9931b53b1c4c42f98f3ab7cb890d5b297ef79ba26c7e6"}
Oct 31 16:04:08.876612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.876163    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"3a4509c8f8084cfc24b5a1c338594578ca2c358c8eb065d13a174bda88030db2"}
Oct 31 16:04:08.876612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.876189    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"21ca5f0ab368e000e30d5d7994f9b02ebb4b293a0f018aefee5174fc5da32803"}
Oct 31 16:04:08.876612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.876220    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"07def99579ceb80fa67c76bf789d82c3a11d8c5878882cf8a03442bca5911f78"}
Oct 31 16:04:08.876612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.876250    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"d63b634f12981020dfc626b582918ab64a5a08ef6882d020eaac231094ca5f4d"}
Oct 31 16:04:08.876612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.876274    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b09ab819-dfb8-46c1-a23e-53dcd927f337","Type":"ContainerStarted","Data":"baa7bfd72f1bf0bae80dc7dd31bf9210b17692a6c0113d2d4c2cdc051d62e8b1"}
Oct 31 16:04:08.993670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:08.993611    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.975928783 podStartE2EDuration="2.993591554s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.662771ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:04:08.992353071 +0000 UTC m=+92.692282414" watchObservedRunningTime="2026-10-31 16:04:08.993591554 +0000 UTC m=+92.693520869"
Oct 31 16:04:11.395271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:04:11.395233    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:04:24.005616 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:24.005568    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:04:54.005256 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:04:54.005215    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:05:06.395289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:05:06.395243    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:05:06.410591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:05:06.410564    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:05:07.103347 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:05:07.103318    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 16:05:24.003445 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:05:24.003400    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:05:54.002359 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:05:54.002320    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:06:24.002999 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:06:24.002918    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:06:51.046987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.046954    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"]
Oct 31 16:06:51.050415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.050399    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.170098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.170065    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"]
Oct 31 16:06:51.187777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.187744    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 16:06:51.187777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.187761    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 16:06:51.187949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.187862    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zc6f5\""
Oct 31 16:06:51.187949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.187915    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 16:06:51.188493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.188476    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 16:06:51.188581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.188481    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 16:06:51.216229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.216201    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.216363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.216241    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.216425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.216403    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.216495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.216446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.216539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.216523    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.216583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.216545    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrr2\" (UniqueName: \"kubernetes.io/projected/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-kube-api-access-hsrr2\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317155    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317215    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrr2\" (UniqueName: \"kubernetes.io/projected/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-kube-api-access-hsrr2\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317240    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317261    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317325    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.317789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.317762    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.319704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.319674    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.319704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.319699    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.319877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.319730    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.319877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.319831    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.372995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.372970    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrr2\" (UniqueName: \"kubernetes.io/projected/8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526-kube-api-access-hsrr2\") pod \"opendatahub-operator-controller-manager-54f884c9ff-rvzzv\" (UID: \"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:51.664351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:51.664255    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:52.224004 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:06:52.223975    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef07dc1_a5a4_48a6_9d39_6e5b84a2b526.slice/crio-b523671bc7f8de7b27a706aaa9ca8d15f72f221a8505728b1c91ea72316d737d WatchSource:0}: Error finding container b523671bc7f8de7b27a706aaa9ca8d15f72f221a8505728b1c91ea72316d737d: Status 404 returned error can't find the container with id b523671bc7f8de7b27a706aaa9ca8d15f72f221a8505728b1c91ea72316d737d
Oct 31 16:06:52.269054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:52.269025    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"]
Oct 31 16:06:52.456292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:52.456259    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv" event={"ID":"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526","Type":"ContainerStarted","Data":"b523671bc7f8de7b27a706aaa9ca8d15f72f221a8505728b1c91ea72316d737d"}
Oct 31 16:06:54.015392 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:06:54.013089    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:06:56.475957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.475921    4519 generic.go:356] "Generic (PLEG): container finished" podID="8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526" containerID="4999b382b4bd37abf8d979dd0a47a61a9ec4ccd67168153e066f61b04bca2ea9" exitCode=0
Oct 31 16:06:56.476393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.476010    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv" event={"ID":"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526","Type":"ContainerDied","Data":"4999b382b4bd37abf8d979dd0a47a61a9ec4ccd67168153e066f61b04bca2ea9"}
Oct 31 16:06:56.621902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.621830    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"]
Oct 31 16:06:56.625074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.625058    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.666775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.666748    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"]
Oct 31 16:06:56.676701 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.676671    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-rttcz\""
Oct 31 16:06:56.676849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.676712    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 16:06:56.676849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.676709    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:06:56.763994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.763960    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c077f7-235a-4d15-951f-a4d6aeb8ebe1-tmp\") pod \"jobset-operator-674cd59dc5-hfwkk\" (UID: \"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.763994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.763996    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlzh\" (UniqueName: \"kubernetes.io/projected/d5c077f7-235a-4d15-951f-a4d6aeb8ebe1-kube-api-access-dxlzh\") pod \"jobset-operator-674cd59dc5-hfwkk\" (UID: \"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.865226 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.865194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c077f7-235a-4d15-951f-a4d6aeb8ebe1-tmp\") pod \"jobset-operator-674cd59dc5-hfwkk\" (UID: \"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.865226 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.865228    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlzh\" (UniqueName: \"kubernetes.io/projected/d5c077f7-235a-4d15-951f-a4d6aeb8ebe1-kube-api-access-dxlzh\") pod \"jobset-operator-674cd59dc5-hfwkk\" (UID: \"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.865636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.865613    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c077f7-235a-4d15-951f-a4d6aeb8ebe1-tmp\") pod \"jobset-operator-674cd59dc5-hfwkk\" (UID: \"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.919176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.919106    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlzh\" (UniqueName: \"kubernetes.io/projected/d5c077f7-235a-4d15-951f-a4d6aeb8ebe1-kube-api-access-dxlzh\") pod \"jobset-operator-674cd59dc5-hfwkk\" (UID: \"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:56.952951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:56.952924    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"
Oct 31 16:06:57.314726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:57.314692    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk"]
Oct 31 16:06:57.317490 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:06:57.317444    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5c077f7_235a_4d15_951f_a4d6aeb8ebe1.slice/crio-ea42ce054aacc647e67860f3d4c097d8a2e40e8a368919a458d6ef0b685c4e26 WatchSource:0}: Error finding container ea42ce054aacc647e67860f3d4c097d8a2e40e8a368919a458d6ef0b685c4e26: Status 404 returned error can't find the container with id ea42ce054aacc647e67860f3d4c097d8a2e40e8a368919a458d6ef0b685c4e26
Oct 31 16:06:57.481866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:57.481816    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk" event={"ID":"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1","Type":"ContainerStarted","Data":"ea42ce054aacc647e67860f3d4c097d8a2e40e8a368919a458d6ef0b685c4e26"}
Oct 31 16:06:57.484268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:57.484237    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv" event={"ID":"8ef07dc1-a5a4-48a6-9d39-6e5b84a2b526","Type":"ContainerStarted","Data":"8e77f4ae8e864415e1744f034ccb873bbce6fed03b394917d1b5191a08d1dd2a"}
Oct 31 16:06:57.484390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:57.484374    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:06:57.665066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:06:57.664941    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv" podStartSLOduration=2.776150581 podStartE2EDuration="7.664924261s" totalImagesPullingTime="4.437880354s" totalInitContainerRuntime="450.893326ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:06:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:57.615639019 +0000 UTC m=+261.315568329" watchObservedRunningTime="2026-10-31 16:06:57.664924261 +0000 UTC m=+261.364853578"
Oct 31 16:07:02.507639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:02.507605    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk" event={"ID":"d5c077f7-235a-4d15-951f-a4d6aeb8ebe1","Type":"ContainerStarted","Data":"ec2345f421004dac8e1681a947785a3d52690dd5bd51229cea092a14a7ea0fc2"}
Oct 31 16:07:02.616969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:02.616919    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hfwkk" podStartSLOduration=1.827454959 podStartE2EDuration="6.616905407s" totalImagesPullingTime="4.789450448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:02.616320204 +0000 UTC m=+266.316249516" watchObservedRunningTime="2026-10-31 16:07:02.616905407 +0000 UTC m=+266.316834718"
Oct 31 16:07:08.490621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:08.490593    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-rvzzv"
Oct 31 16:07:21.297393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.297360    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"]
Oct 31 16:07:21.300835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.300819    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:21.357820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.357795    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 16:07:21.372604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.372578    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzbl\" (UniqueName: \"kubernetes.io/projected/db6c0f50-71af-4052-925d-1dc7fb1d0327-kube-api-access-5rzbl\") pod \"dns-operator-controller-manager-8557cbdb84-hs8rg\" (UID: \"db6c0f50-71af-4052-925d-1dc7fb1d0327\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:21.404449 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.404425    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 16:07:21.404595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.404518    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 16:07:21.404595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.404527    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-prc5x\""
Oct 31 16:07:21.405774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.405759    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"]
Oct 31 16:07:21.473989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.473953    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzbl\" (UniqueName: \"kubernetes.io/projected/db6c0f50-71af-4052-925d-1dc7fb1d0327-kube-api-access-5rzbl\") pod \"dns-operator-controller-manager-8557cbdb84-hs8rg\" (UID: \"db6c0f50-71af-4052-925d-1dc7fb1d0327\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:21.529356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.529320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzbl\" (UniqueName: \"kubernetes.io/projected/db6c0f50-71af-4052-925d-1dc7fb1d0327-kube-api-access-5rzbl\") pod \"dns-operator-controller-manager-8557cbdb84-hs8rg\" (UID: \"db6c0f50-71af-4052-925d-1dc7fb1d0327\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:21.613539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.613445    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:21.981729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:21.981681    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"]
Oct 31 16:07:21.983525 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:21.983498    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6c0f50_71af_4052_925d_1dc7fb1d0327.slice/crio-9ab1fb9ceefe82ef21062b881c9c2ca845d46b715ae8637c2d2dc3aabba4626c WatchSource:0}: Error finding container 9ab1fb9ceefe82ef21062b881c9c2ca845d46b715ae8637c2d2dc3aabba4626c: Status 404 returned error can't find the container with id 9ab1fb9ceefe82ef21062b881c9c2ca845d46b715ae8637c2d2dc3aabba4626c
Oct 31 16:07:22.584877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:22.584845    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg" event={"ID":"db6c0f50-71af-4052-925d-1dc7fb1d0327","Type":"ContainerStarted","Data":"9ab1fb9ceefe82ef21062b881c9c2ca845d46b715ae8637c2d2dc3aabba4626c"}
Oct 31 16:07:23.405477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.405421    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-2cx6v"]
Oct 31 16:07:23.409987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.409971    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.462597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.462563    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 16:07:23.463822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.463801    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2cx6v"]
Oct 31 16:07:23.491225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.491197    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp64j\" (UniqueName: \"kubernetes.io/projected/35a8c308-796f-4271-bb8f-23470b937a46-kube-api-access-gp64j\") pod \"cert-manager-597b95cb84-2cx6v\" (UID: \"35a8c308-796f-4271-bb8f-23470b937a46\") " pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.491334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.491229    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35a8c308-796f-4271-bb8f-23470b937a46-bound-sa-token\") pod \"cert-manager-597b95cb84-2cx6v\" (UID: \"35a8c308-796f-4271-bb8f-23470b937a46\") " pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.508462 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.508438    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qnhg9\""
Oct 31 16:07:23.508557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.508438    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 16:07:23.592277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.592246    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp64j\" (UniqueName: \"kubernetes.io/projected/35a8c308-796f-4271-bb8f-23470b937a46-kube-api-access-gp64j\") pod \"cert-manager-597b95cb84-2cx6v\" (UID: \"35a8c308-796f-4271-bb8f-23470b937a46\") " pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.592748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.592286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35a8c308-796f-4271-bb8f-23470b937a46-bound-sa-token\") pod \"cert-manager-597b95cb84-2cx6v\" (UID: \"35a8c308-796f-4271-bb8f-23470b937a46\") " pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.651287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.651256    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35a8c308-796f-4271-bb8f-23470b937a46-bound-sa-token\") pod \"cert-manager-597b95cb84-2cx6v\" (UID: \"35a8c308-796f-4271-bb8f-23470b937a46\") " pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.651431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.651380    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp64j\" (UniqueName: \"kubernetes.io/projected/35a8c308-796f-4271-bb8f-23470b937a46-kube-api-access-gp64j\") pod \"cert-manager-597b95cb84-2cx6v\" (UID: \"35a8c308-796f-4271-bb8f-23470b937a46\") " pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:23.723541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:23.723506    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2cx6v"
Oct 31 16:07:24.002707 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:07:24.002630    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:07:24.086470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:24.086428    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2cx6v"]
Oct 31 16:07:24.088641 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:24.088613    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35a8c308_796f_4271_bb8f_23470b937a46.slice/crio-bfe3b04451ac1b18e414dbb4d6182c6764af253e4b528825e54b729530bb88b9 WatchSource:0}: Error finding container bfe3b04451ac1b18e414dbb4d6182c6764af253e4b528825e54b729530bb88b9: Status 404 returned error can't find the container with id bfe3b04451ac1b18e414dbb4d6182c6764af253e4b528825e54b729530bb88b9
Oct 31 16:07:24.594512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:24.594473    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2cx6v" event={"ID":"35a8c308-796f-4271-bb8f-23470b937a46","Type":"ContainerStarted","Data":"bfe3b04451ac1b18e414dbb4d6182c6764af253e4b528825e54b729530bb88b9"}
Oct 31 16:07:25.162637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.162604    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"]
Oct 31 16:07:25.166564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.166537    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.224981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.224951    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 16:07:25.225192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.225021    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 16:07:25.227357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.227324    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"]
Oct 31 16:07:25.252490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.252448    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 16:07:25.268735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.268713    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cc559\""
Oct 31 16:07:25.268881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.268719    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 16:07:25.309337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.309304    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5b308c3-b987-4fc1-a847-94a8fa7afac0-tmp\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.309544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.309462    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b308c3-b987-4fc1-a847-94a8fa7afac0-tls-certs\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.309616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.309541    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrffv\" (UniqueName: \"kubernetes.io/projected/c5b308c3-b987-4fc1-a847-94a8fa7afac0-kube-api-access-qrffv\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.410500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.410451    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5b308c3-b987-4fc1-a847-94a8fa7afac0-tmp\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.410653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.410551    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b308c3-b987-4fc1-a847-94a8fa7afac0-tls-certs\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.410653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.410586    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrffv\" (UniqueName: \"kubernetes.io/projected/c5b308c3-b987-4fc1-a847-94a8fa7afac0-kube-api-access-qrffv\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.413711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.413650    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5b308c3-b987-4fc1-a847-94a8fa7afac0-tmp\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.413866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.413751    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b308c3-b987-4fc1-a847-94a8fa7afac0-tls-certs\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.465516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.465438    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrffv\" (UniqueName: \"kubernetes.io/projected/c5b308c3-b987-4fc1-a847-94a8fa7afac0-kube-api-access-qrffv\") pod \"kube-auth-proxy-744f97f697-vjlc2\" (UID: \"c5b308c3-b987-4fc1-a847-94a8fa7afac0\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.480584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.480556    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"
Oct 31 16:07:25.857328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:25.857293    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-744f97f697-vjlc2"]
Oct 31 16:07:25.860872 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:25.860836    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b308c3_b987_4fc1_a847_94a8fa7afac0.slice/crio-487f7a8e55b141393cfd418fd5004d3c79c5a24b8b846fd0099dd6834f6f2799 WatchSource:0}: Error finding container 487f7a8e55b141393cfd418fd5004d3c79c5a24b8b846fd0099dd6834f6f2799: Status 404 returned error can't find the container with id 487f7a8e55b141393cfd418fd5004d3c79c5a24b8b846fd0099dd6834f6f2799
Oct 31 16:07:26.604907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:26.604855    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2" event={"ID":"c5b308c3-b987-4fc1-a847-94a8fa7afac0","Type":"ContainerStarted","Data":"487f7a8e55b141393cfd418fd5004d3c79c5a24b8b846fd0099dd6834f6f2799"}
Oct 31 16:07:27.616420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:27.616385    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg" event={"ID":"db6c0f50-71af-4052-925d-1dc7fb1d0327","Type":"ContainerStarted","Data":"1a8635be645bc7783d389ab321cd74e053382f729e0363b5205312155ffcaa7a"}
Oct 31 16:07:27.616905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:27.616496    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:27.724999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:27.724910    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg" podStartSLOduration=1.8374786749999998 podStartE2EDuration="6.724892186s" totalImagesPullingTime="4.887413511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:27.724752057 +0000 UTC m=+291.424681367" watchObservedRunningTime="2026-10-31 16:07:27.724892186 +0000 UTC m=+291.424821500"
Oct 31 16:07:29.627855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:29.627816    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2cx6v" event={"ID":"35a8c308-796f-4271-bb8f-23470b937a46","Type":"ContainerStarted","Data":"5a6b53e58817059f6548a846f14b5521696d14e89f478209b674f6379ac485cb"}
Oct 31 16:07:29.629305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:29.629280    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2" event={"ID":"c5b308c3-b987-4fc1-a847-94a8fa7afac0","Type":"ContainerStarted","Data":"aca814ce1e35d39ac0e0b784d44528e4fd39b4510e146589cdeb292c965625bd"}
Oct 31 16:07:29.735953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:29.735904    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-2cx6v" podStartSLOduration=1.963969785 podStartE2EDuration="6.735890763s" totalImagesPullingTime="4.771920978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:29.73528072 +0000 UTC m=+293.435210033" watchObservedRunningTime="2026-10-31 16:07:29.735890763 +0000 UTC m=+293.435820074"
Oct 31 16:07:29.871339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:29.871284    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-744f97f697-vjlc2" podStartSLOduration=1.5412731389999998 podStartE2EDuration="4.871267627s" totalImagesPullingTime="3.329994488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:29.86973093 +0000 UTC m=+293.569660242" watchObservedRunningTime="2026-10-31 16:07:29.871267627 +0000 UTC m=+293.571196940"
Oct 31 16:07:30.414581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.414546    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:30.417980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.417960    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.470855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.470808    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 16:07:30.518015 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.517983    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4r6xh\""
Oct 31 16:07:30.519938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.519900    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:30.560291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.560261    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file\") pod \"limitador-limitador-74dfcfb894-8g874\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.560446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.560322    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnkb\" (UniqueName: \"kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb\") pod \"limitador-limitador-74dfcfb894-8g874\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.661248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.661215    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file\") pod \"limitador-limitador-74dfcfb894-8g874\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.661635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.661282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnkb\" (UniqueName: \"kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb\") pod \"limitador-limitador-74dfcfb894-8g874\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.661954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.661934    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file\") pod \"limitador-limitador-74dfcfb894-8g874\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.666498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.666434    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:30.715136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.715106    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnkb\" (UniqueName: \"kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb\") pod \"limitador-limitador-74dfcfb894-8g874\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:30.731006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:30.730981    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:31.096175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.096150    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:31.097256 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:31.097229    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa158a17_6373_4587_90b4_f22acaf1a0ca.slice/crio-7512aa5a434c368393fee2871a15aed92e48c42ca84d2e6dfd9d3250c2f65c89 WatchSource:0}: Error finding container 7512aa5a434c368393fee2871a15aed92e48c42ca84d2e6dfd9d3250c2f65c89: Status 404 returned error can't find the container with id 7512aa5a434c368393fee2871a15aed92e48c42ca84d2e6dfd9d3250c2f65c89
Oct 31 16:07:31.558176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.558143    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2r4j4"]
Oct 31 16:07:31.571136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.571111    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:31.614007 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.613974    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2r4j4"]
Oct 31 16:07:31.620680 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.620649    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hxc6d\""
Oct 31 16:07:31.639706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.639673    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874" event={"ID":"aa158a17-6373-4587-90b4-f22acaf1a0ca","Type":"ContainerStarted","Data":"7512aa5a434c368393fee2871a15aed92e48c42ca84d2e6dfd9d3250c2f65c89"}
Oct 31 16:07:31.671743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.671714    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfwr\" (UniqueName: \"kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr\") pod \"authorino-57f7f8659f-2r4j4\" (UID: \"04e91dc6-2d51-4c55-8a77-053a4649c574\") " pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:31.772905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.772867    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfwr\" (UniqueName: \"kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr\") pod \"authorino-57f7f8659f-2r4j4\" (UID: \"04e91dc6-2d51-4c55-8a77-053a4649c574\") " pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:31.827242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.827174    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfwr\" (UniqueName: \"kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr\") pod \"authorino-57f7f8659f-2r4j4\" (UID: \"04e91dc6-2d51-4c55-8a77-053a4649c574\") " pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:31.883037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:31.883002    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:32.242909 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:32.242883    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2r4j4"]
Oct 31 16:07:32.244664 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:32.244635    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04e91dc6_2d51_4c55_8a77_053a4649c574.slice/crio-f03dbdd034604cdad9556799766bcb0082cefe6d5977eb91be73ec1333e2ff75 WatchSource:0}: Error finding container f03dbdd034604cdad9556799766bcb0082cefe6d5977eb91be73ec1333e2ff75: Status 404 returned error can't find the container with id f03dbdd034604cdad9556799766bcb0082cefe6d5977eb91be73ec1333e2ff75
Oct 31 16:07:32.645594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:32.645510    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2r4j4" event={"ID":"04e91dc6-2d51-4c55-8a77-053a4649c574","Type":"ContainerStarted","Data":"f03dbdd034604cdad9556799766bcb0082cefe6d5977eb91be73ec1333e2ff75"}
Oct 31 16:07:33.180789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.180754    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:07:33.185381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.185362    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.259262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.259090    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:07:33.286478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.286426    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3cf71a8-4583-48a6-990d-52e3afe1eeea-config-file\") pod \"limitador-limitador-7c9b87d574-jwxmj\" (UID: \"d3cf71a8-4583-48a6-990d-52e3afe1eeea\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.286675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.286509    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47hx\" (UniqueName: \"kubernetes.io/projected/d3cf71a8-4583-48a6-990d-52e3afe1eeea-kube-api-access-l47hx\") pod \"limitador-limitador-7c9b87d574-jwxmj\" (UID: \"d3cf71a8-4583-48a6-990d-52e3afe1eeea\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.387602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.387565    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3cf71a8-4583-48a6-990d-52e3afe1eeea-config-file\") pod \"limitador-limitador-7c9b87d574-jwxmj\" (UID: \"d3cf71a8-4583-48a6-990d-52e3afe1eeea\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.387797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.387622    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47hx\" (UniqueName: \"kubernetes.io/projected/d3cf71a8-4583-48a6-990d-52e3afe1eeea-kube-api-access-l47hx\") pod \"limitador-limitador-7c9b87d574-jwxmj\" (UID: \"d3cf71a8-4583-48a6-990d-52e3afe1eeea\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.388264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.388238    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3cf71a8-4583-48a6-990d-52e3afe1eeea-config-file\") pod \"limitador-limitador-7c9b87d574-jwxmj\" (UID: \"d3cf71a8-4583-48a6-990d-52e3afe1eeea\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.443328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.443259    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47hx\" (UniqueName: \"kubernetes.io/projected/d3cf71a8-4583-48a6-990d-52e3afe1eeea-kube-api-access-l47hx\") pod \"limitador-limitador-7c9b87d574-jwxmj\" (UID: \"d3cf71a8-4583-48a6-990d-52e3afe1eeea\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.497908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.497873    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:33.509296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.509271    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:07:33.860730 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:33.860705    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:07:33.862731 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:33.862695    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cf71a8_4583_48a6_990d_52e3afe1eeea.slice/crio-fc0e7fea2a9a669cdf43dc3a2a0ff55aa0072b8741a303557ad9a975cf07fc1e WatchSource:0}: Error finding container fc0e7fea2a9a669cdf43dc3a2a0ff55aa0072b8741a303557ad9a975cf07fc1e: Status 404 returned error can't find the container with id fc0e7fea2a9a669cdf43dc3a2a0ff55aa0072b8741a303557ad9a975cf07fc1e
Oct 31 16:07:34.654655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:34.654618    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj" event={"ID":"d3cf71a8-4583-48a6-990d-52e3afe1eeea","Type":"ContainerStarted","Data":"fc0e7fea2a9a669cdf43dc3a2a0ff55aa0072b8741a303557ad9a975cf07fc1e"}
Oct 31 16:07:35.660569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:35.660534    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2r4j4" event={"ID":"04e91dc6-2d51-4c55-8a77-053a4649c574","Type":"ContainerStarted","Data":"277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e"}
Oct 31 16:07:35.781921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:35.781850    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2r4j4" podStartSLOduration=1.829751143 podStartE2EDuration="4.781830362s" totalImagesPullingTime="2.952079219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:35.781109707 +0000 UTC m=+299.481039020" watchObservedRunningTime="2026-10-31 16:07:35.781830362 +0000 UTC m=+299.481759696"
Oct 31 16:07:36.667797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:36.667702    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj" event={"ID":"d3cf71a8-4583-48a6-990d-52e3afe1eeea","Type":"ContainerStarted","Data":"ec65e5c305051234efdac3c9c88e135d11684d84c6fce2fd611291e70ad28989"}
Oct 31 16:07:36.775346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:36.775291    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj" podStartSLOduration=1.409642559 podStartE2EDuration="3.775277306s" totalImagesPullingTime="2.365634747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:36.774627285 +0000 UTC m=+300.474556596" watchObservedRunningTime="2026-10-31 16:07:36.775277306 +0000 UTC m=+300.475206616"
Oct 31 16:07:37.359331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:37.359298    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2r4j4"]
Oct 31 16:07:37.674027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:37.673927    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874" event={"ID":"aa158a17-6373-4587-90b4-f22acaf1a0ca","Type":"ContainerStarted","Data":"7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696"}
Oct 31 16:07:37.674027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:37.673980    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2r4j4" podUID="04e91dc6-2d51-4c55-8a77-053a4649c574" containerName="authorino" containerID="cri-o://277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e" gracePeriod=30
Oct 31 16:07:37.674538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:37.674309    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:37.782709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:37.782658    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874" podStartSLOduration=1.580606358 podStartE2EDuration="7.782644577s" totalImagesPullingTime="6.202038219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:37.782261239 +0000 UTC m=+301.482190549" watchObservedRunningTime="2026-10-31 16:07:37.782644577 +0000 UTC m=+301.482573912"
Oct 31 16:07:37.914151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:37.914121    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:38.032751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.032721    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr\" (UniqueName: \"kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr\") pod \"04e91dc6-2d51-4c55-8a77-053a4649c574\" (UID: \"04e91dc6-2d51-4c55-8a77-053a4649c574\") "
Oct 31 16:07:38.035127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.035090    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr" pod "04e91dc6-2d51-4c55-8a77-053a4649c574" (UID: "04e91dc6-2d51-4c55-8a77-053a4649c574"). InnerVolumeSpecName "kube-api-access-jvfwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:07:38.133409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.133367    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvfwr\" (UniqueName: \"kubernetes.io/projected/04e91dc6-2d51-4c55-8a77-053a4649c574-kube-api-access-jvfwr\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:07:38.238624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.238589    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"]
Oct 31 16:07:38.239008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.238995    4519 cpu_manager.go:414] "containerMap: removing container" podUID="04e91dc6-2d51-4c55-8a77-053a4649c574" containerName="authorino"
Oct 31 16:07:38.239062 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.239010    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e91dc6-2d51-4c55-8a77-053a4649c574" containerName="authorino"
Oct 31 16:07:38.239109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.239073    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e91dc6-2d51-4c55-8a77-053a4649c574" containerName="authorino"
Oct 31 16:07:38.242098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.242081    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.295696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.295588    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 16:07:38.295696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.295594    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 16:07:38.297593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.297570    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"]
Oct 31 16:07:38.344256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.344220    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 16:07:38.387388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.387359    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-978zt\""
Oct 31 16:07:38.436260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.436227    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00b120bd-2b41-470e-bf2c-2a575adaba5c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.436398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.436344    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7rn\" (UniqueName: \"kubernetes.io/projected/00b120bd-2b41-470e-bf2c-2a575adaba5c-kube-api-access-pw7rn\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.436478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.436421    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b120bd-2b41-470e-bf2c-2a575adaba5c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.436478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.436450    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00b120bd-2b41-470e-bf2c-2a575adaba5c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.537254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.537218    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7rn\" (UniqueName: \"kubernetes.io/projected/00b120bd-2b41-470e-bf2c-2a575adaba5c-kube-api-access-pw7rn\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.537484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.537262    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b120bd-2b41-470e-bf2c-2a575adaba5c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.537484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.537286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00b120bd-2b41-470e-bf2c-2a575adaba5c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.537484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.537326    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00b120bd-2b41-470e-bf2c-2a575adaba5c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.537729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.537709    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00b120bd-2b41-470e-bf2c-2a575adaba5c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.537787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.537708    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b120bd-2b41-470e-bf2c-2a575adaba5c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.538117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.538099    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00b120bd-2b41-470e-bf2c-2a575adaba5c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.591024 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.590960    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7rn\" (UniqueName: \"kubernetes.io/projected/00b120bd-2b41-470e-bf2c-2a575adaba5c-kube-api-access-pw7rn\") pod \"kserve-module-controller-manager-7566cf5fc8-9w5mq\" (UID: \"00b120bd-2b41-470e-bf2c-2a575adaba5c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:38.624709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.624682    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hs8rg"
Oct 31 16:07:38.678843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.678808    4519 generic.go:356] "Generic (PLEG): container finished" podID="04e91dc6-2d51-4c55-8a77-053a4649c574" containerID="277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e" exitCode=0
Oct 31 16:07:38.679300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.678852    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2r4j4"
Oct 31 16:07:38.679300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.678897    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2r4j4" event={"ID":"04e91dc6-2d51-4c55-8a77-053a4649c574","Type":"ContainerDied","Data":"277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e"}
Oct 31 16:07:38.679300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.678949    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2r4j4" event={"ID":"04e91dc6-2d51-4c55-8a77-053a4649c574","Type":"ContainerDied","Data":"f03dbdd034604cdad9556799766bcb0082cefe6d5977eb91be73ec1333e2ff75"}
Oct 31 16:07:38.679300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.678972    4519 scope.go:122] "RemoveContainer" containerID="277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e"
Oct 31 16:07:38.679300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.679219    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:38.686892 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.686876    4519 scope.go:122] "RemoveContainer" containerID="277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e"
Oct 31 16:07:38.687156 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:07:38.687138    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e\": container with ID starting with 277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e not found: ID does not exist" containerID="277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e"
Oct 31 16:07:38.687194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.687168    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e"} err="failed to get container status \"277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e\": rpc error: code = NotFound desc = could not find container \"277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e\": container with ID starting with 277d4887581e403206d2714e28fd9c9e2cd0f25dabd4a37729b7e48adb828d6e not found: ID does not exist"
Oct 31 16:07:38.838721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.838686    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2r4j4"]
Oct 31 16:07:38.840437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.840410    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2r4j4"]
Oct 31 16:07:38.854610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:38.854552    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:39.272286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:39.272249    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"]
Oct 31 16:07:39.305624 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:07:39.305594    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b120bd_2b41_470e_bf2c_2a575adaba5c.slice/crio-c5f44ef80746f59faaa350b8a075609c3f23ec0e23ec0f8b1ced0d332fe8556c WatchSource:0}: Error finding container c5f44ef80746f59faaa350b8a075609c3f23ec0e23ec0f8b1ced0d332fe8556c: Status 404 returned error can't find the container with id c5f44ef80746f59faaa350b8a075609c3f23ec0e23ec0f8b1ced0d332fe8556c
Oct 31 16:07:39.307743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:39.307723    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:07:39.685725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:39.685639    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq" event={"ID":"00b120bd-2b41-470e-bf2c-2a575adaba5c","Type":"ContainerStarted","Data":"c5f44ef80746f59faaa350b8a075609c3f23ec0e23ec0f8b1ced0d332fe8556c"}
Oct 31 16:07:40.436529 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:40.436499    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e91dc6-2d51-4c55-8a77-053a4649c574" path="/var/lib/kubelet/pods/04e91dc6-2d51-4c55-8a77-053a4649c574/volumes"
Oct 31 16:07:41.696371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:41.696284    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq" event={"ID":"00b120bd-2b41-470e-bf2c-2a575adaba5c","Type":"ContainerStarted","Data":"ecdc0b528979be98282537de34c6ca9d12361054eb342f47b25acb198abee150"}
Oct 31 16:07:41.696750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:41.696479    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:07:41.854524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:41.854467    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq" podStartSLOduration=1.7801386030000002 podStartE2EDuration="3.854441987s" totalImagesPullingTime="2.074303384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:41.85275792 +0000 UTC m=+305.552687230" watchObservedRunningTime="2026-10-31 16:07:41.854441987 +0000 UTC m=+305.554371297"
Oct 31 16:07:48.679850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:48.679773    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"
Oct 31 16:07:48.835289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:48.835261    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:48.835501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:48.835475    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874" podUID="aa158a17-6373-4587-90b4-f22acaf1a0ca" containerName="limitador" containerID="cri-o://7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696" gracePeriod=30
Oct 31 16:07:48.837533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:48.837508    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:49.374037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.374015    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:49.413974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.413899    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file\" (UniqueName: \"kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file\") pod \"aa158a17-6373-4587-90b4-f22acaf1a0ca\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") "
Oct 31 16:07:49.413974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.413945    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb\" (UniqueName: \"kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb\") pod \"aa158a17-6373-4587-90b4-f22acaf1a0ca\" (UID: \"aa158a17-6373-4587-90b4-f22acaf1a0ca\") "
Oct 31 16:07:49.414255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.414233    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file" pod "aa158a17-6373-4587-90b4-f22acaf1a0ca" (UID: "aa158a17-6373-4587-90b4-f22acaf1a0ca"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:07:49.415962 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.415936    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb" pod "aa158a17-6373-4587-90b4-f22acaf1a0ca" (UID: "aa158a17-6373-4587-90b4-f22acaf1a0ca"). InnerVolumeSpecName "kube-api-access-gmnkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:07:49.515378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.515343    4519 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa158a17-6373-4587-90b4-f22acaf1a0ca-config-file\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:07:49.515378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.515373    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmnkb\" (UniqueName: \"kubernetes.io/projected/aa158a17-6373-4587-90b4-f22acaf1a0ca-kube-api-access-gmnkb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:07:49.730409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.730373    4519 generic.go:356] "Generic (PLEG): container finished" podID="aa158a17-6373-4587-90b4-f22acaf1a0ca" containerID="7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696" exitCode=0
Oct 31 16:07:49.730879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.730425    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874"
Oct 31 16:07:49.730879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.730469    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874" event={"ID":"aa158a17-6373-4587-90b4-f22acaf1a0ca","Type":"ContainerDied","Data":"7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696"}
Oct 31 16:07:49.730879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.730540    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-8g874" event={"ID":"aa158a17-6373-4587-90b4-f22acaf1a0ca","Type":"ContainerDied","Data":"7512aa5a434c368393fee2871a15aed92e48c42ca84d2e6dfd9d3250c2f65c89"}
Oct 31 16:07:49.730879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.730565    4519 scope.go:122] "RemoveContainer" containerID="7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696"
Oct 31 16:07:49.739734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.739713    4519 scope.go:122] "RemoveContainer" containerID="7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696"
Oct 31 16:07:49.740034 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:07:49.740014    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696\": container with ID starting with 7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696 not found: ID does not exist" containerID="7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696"
Oct 31 16:07:49.740101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.740047    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696"} err="failed to get container status \"7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696\": rpc error: code = NotFound desc = could not find container \"7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696\": container with ID starting with 7322741084c8cb355382f37b896c0f04a98352ed30b6e920021eee7e93f97696 not found: ID does not exist"
Oct 31 16:07:49.888751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.888717    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:49.931352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:49.931317    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-8g874"]
Oct 31 16:07:50.436932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:07:50.436902    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa158a17-6373-4587-90b4-f22acaf1a0ca" path="/var/lib/kubelet/pods/aa158a17-6373-4587-90b4-f22acaf1a0ca/volumes"
Oct 31 16:07:54.003866 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:07:54.003827    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:08:00.477239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.477197    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"]
Oct 31 16:08:00.477792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.477735    4519 cpu_manager.go:414] "containerMap: removing container" podUID="aa158a17-6373-4587-90b4-f22acaf1a0ca" containerName="limitador"
Oct 31 16:08:00.477792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.477754    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa158a17-6373-4587-90b4-f22acaf1a0ca" containerName="limitador"
Oct 31 16:08:00.477931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.477917    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa158a17-6373-4587-90b4-f22acaf1a0ca" containerName="limitador"
Oct 31 16:08:00.485447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.485424    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.513022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.512990    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71eb9ea0-765d-4684-b4ca-803942305b02-cert\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.513022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.513024    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx97t\" (UniqueName: \"kubernetes.io/projected/71eb9ea0-765d-4684-b4ca-803942305b02-kube-api-access-jx97t\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.513192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.513109    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71eb9ea0-765d-4684-b4ca-803942305b02-metrics-certs\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.513192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.513152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71eb9ea0-765d-4684-b4ca-803942305b02-manager-config\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.532919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.532890    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"]
Oct 31 16:08:00.578573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.578538    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 16:08:00.578738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.578664    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 16:08:00.581658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.581644    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 16:08:00.581710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.581646    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-42k98\""
Oct 31 16:08:00.613877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.613840    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71eb9ea0-765d-4684-b4ca-803942305b02-metrics-certs\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.614058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.613905    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71eb9ea0-765d-4684-b4ca-803942305b02-manager-config\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.614058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.613957    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71eb9ea0-765d-4684-b4ca-803942305b02-cert\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.614058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.613981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx97t\" (UniqueName: \"kubernetes.io/projected/71eb9ea0-765d-4684-b4ca-803942305b02-kube-api-access-jx97t\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.614595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.614575    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71eb9ea0-765d-4684-b4ca-803942305b02-manager-config\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.616475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.616422    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71eb9ea0-765d-4684-b4ca-803942305b02-cert\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.616571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.616512    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71eb9ea0-765d-4684-b4ca-803942305b02-metrics-certs\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.675370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.675329    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx97t\" (UniqueName: \"kubernetes.io/projected/71eb9ea0-765d-4684-b4ca-803942305b02-kube-api-access-jx97t\") pod \"jobset-controller-manager-5d6478c64c-rmrqc\" (UID: \"71eb9ea0-765d-4684-b4ca-803942305b02\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:00.798221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:00.798138    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:01.159176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:01.159139    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"]
Oct 31 16:08:01.162378 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:08:01.162349    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71eb9ea0_765d_4684_b4ca_803942305b02.slice/crio-03338eeabfe16df7ec3fd6ed1f2e64b721d4f5c29a018da58775c1437494ca46 WatchSource:0}: Error finding container 03338eeabfe16df7ec3fd6ed1f2e64b721d4f5c29a018da58775c1437494ca46: Status 404 returned error can't find the container with id 03338eeabfe16df7ec3fd6ed1f2e64b721d4f5c29a018da58775c1437494ca46
Oct 31 16:08:01.783755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:01.783720    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc" event={"ID":"71eb9ea0-765d-4684-b4ca-803942305b02","Type":"ContainerStarted","Data":"03338eeabfe16df7ec3fd6ed1f2e64b721d4f5c29a018da58775c1437494ca46"}
Oct 31 16:08:02.702233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:02.702200    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9w5mq"
Oct 31 16:08:07.811001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:07.810904    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc" event={"ID":"71eb9ea0-765d-4684-b4ca-803942305b02","Type":"ContainerStarted","Data":"96df9e7119fd769fd0c8a068d67f0c974b6cd33f8219804429f16bf1b6dfe513"}
Oct 31 16:08:07.811001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:07.810987    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:07.919836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:07.919786    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc" podStartSLOduration=1.566855564 podStartE2EDuration="7.919772261s" totalImagesPullingTime="6.352916697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:07.918640063 +0000 UTC m=+331.618569373" watchObservedRunningTime="2026-10-31 16:08:07.919772261 +0000 UTC m=+331.619701571"
Oct 31 16:08:18.820928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:08:18.820894    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:08:24.009508 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:08:24.009469    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:08:54.002502 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:08:54.002442    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:09:06.223704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:06.223657    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:09:12.412532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.412496    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tmdqb"]
Oct 31 16:09:12.416290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.416272    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:12.471932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.471905    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tmdqb"]
Oct 31 16:09:12.526005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.525978    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hxc6d\""
Oct 31 16:09:12.557422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.557393    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzv8\" (UniqueName: \"kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8\") pod \"authorino-6f6944bcdd-tmdqb\" (UID: \"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c\") " pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:12.658761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.658735    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzv8\" (UniqueName: \"kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8\") pod \"authorino-6f6944bcdd-tmdqb\" (UID: \"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c\") " pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:12.719728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.719693    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzv8\" (UniqueName: \"kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8\") pod \"authorino-6f6944bcdd-tmdqb\" (UID: \"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c\") " pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:12.728467 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.728431    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:12.774906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:12.774878    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tmdqb"]
Oct 31 16:09:13.088884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:13.088859    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tmdqb"]
Oct 31 16:09:13.091368 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:09:13.091342    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c049041_4cd3_400f_aaf0_1a7c1fc70a8c.slice/crio-ffcc916529b92da9ca9bd33754cace313bc02c13d26adba12b0547cefd6cce44 WatchSource:0}: Error finding container ffcc916529b92da9ca9bd33754cace313bc02c13d26adba12b0547cefd6cce44: Status 404 returned error can't find the container with id ffcc916529b92da9ca9bd33754cace313bc02c13d26adba12b0547cefd6cce44
Oct 31 16:09:14.092389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.092356    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb" event={"ID":"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c","Type":"ContainerStarted","Data":"ffcc916529b92da9ca9bd33754cace313bc02c13d26adba12b0547cefd6cce44"}
Oct 31 16:09:14.884309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.884274    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5754b7df69-sgsls"]
Oct 31 16:09:14.888931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.888911    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:14.946387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.946363    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5754b7df69-sgsls"]
Oct 31 16:09:14.977626 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.977603    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b72w\" (UniqueName: \"kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w\") pod \"authorino-5754b7df69-sgsls\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") " pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:14.977766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.977669    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert\") pod \"authorino-5754b7df69-sgsls\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") " pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:14.998814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:14.998793    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 16:09:15.078567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.078539    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b72w\" (UniqueName: \"kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w\") pod \"authorino-5754b7df69-sgsls\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") " pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:15.078710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.078626    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert\") pod \"authorino-5754b7df69-sgsls\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") " pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:15.080916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.080891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert\") pod \"authorino-5754b7df69-sgsls\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") " pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:15.097840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.097811    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb" event={"ID":"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c","Type":"ContainerStarted","Data":"24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc"}
Oct 31 16:09:15.098189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.097866    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb" podUID="8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" containerName="authorino" containerID="cri-o://24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc" gracePeriod=30
Oct 31 16:09:15.138597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.138538    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b72w\" (UniqueName: \"kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w\") pod \"authorino-5754b7df69-sgsls\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") " pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:15.200610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.200582    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:09:15.245500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.245436    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb" podStartSLOduration=1.353660677 podStartE2EDuration="3.245420702s" totalImagesPullingTime="1.891760025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:15.244377124 +0000 UTC m=+398.944306434" watchObservedRunningTime="2026-10-31 16:09:15.245420702 +0000 UTC m=+398.945350012"
Oct 31 16:09:15.424904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.424880    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:15.562032 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:09:15.561991    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc51d97f_434c_4b55_b6ad_cb8e81c02db4.slice/crio-ad297ab6333e388212ef8b18e5ba183f2d8f52d34dc566acb9d9e7a65abe0fdf WatchSource:0}: Error finding container ad297ab6333e388212ef8b18e5ba183f2d8f52d34dc566acb9d9e7a65abe0fdf: Status 404 returned error can't find the container with id ad297ab6333e388212ef8b18e5ba183f2d8f52d34dc566acb9d9e7a65abe0fdf
Oct 31 16:09:15.583526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.583504    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8\" (UniqueName: \"kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8\") pod \"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c\" (UID: \"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c\") "
Oct 31 16:09:15.585405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.585385    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8" pod "8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" (UID: "8c049041-4cd3-400f-aaf0-1a7c1fc70a8c"). InnerVolumeSpecName "kube-api-access-spzv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:15.599089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.599057    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5754b7df69-sgsls"]
Oct 31 16:09:15.684725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:15.684672    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spzv8\" (UniqueName: \"kubernetes.io/projected/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c-kube-api-access-spzv8\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:09:16.104022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.103985    4519 generic.go:356] "Generic (PLEG): container finished" podID="8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" containerID="24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc" exitCode=0
Oct 31 16:09:16.104488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.104032    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb"
Oct 31 16:09:16.104488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.104066    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb" event={"ID":"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c","Type":"ContainerDied","Data":"24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc"}
Oct 31 16:09:16.104488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.104106    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tmdqb" event={"ID":"8c049041-4cd3-400f-aaf0-1a7c1fc70a8c","Type":"ContainerDied","Data":"ffcc916529b92da9ca9bd33754cace313bc02c13d26adba12b0547cefd6cce44"}
Oct 31 16:09:16.104488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.104130    4519 scope.go:122] "RemoveContainer" containerID="24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc"
Oct 31 16:09:16.105484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.105424    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5754b7df69-sgsls" event={"ID":"bc51d97f-434c-4b55-b6ad-cb8e81c02db4","Type":"ContainerStarted","Data":"ad297ab6333e388212ef8b18e5ba183f2d8f52d34dc566acb9d9e7a65abe0fdf"}
Oct 31 16:09:16.112886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.112869    4519 scope.go:122] "RemoveContainer" containerID="24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc"
Oct 31 16:09:16.113142 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:09:16.113126    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc\": container with ID starting with 24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc not found: ID does not exist" containerID="24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc"
Oct 31 16:09:16.113205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.113148    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc"} err="failed to get container status \"24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc\": rpc error: code = NotFound desc = could not find container \"24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc\": container with ID starting with 24596536b5634c3ebce900aa4f926d12ae30b51e14bd7e8804c888e78b65a0fc not found: ID does not exist"
Oct 31 16:09:16.277507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.277420    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tmdqb"]
Oct 31 16:09:16.331209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.331176    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tmdqb"]
Oct 31 16:09:16.442820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:16.442789    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" path="/var/lib/kubelet/pods/8c049041-4cd3-400f-aaf0-1a7c1fc70a8c/volumes"
Oct 31 16:09:17.110625 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:17.110585    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5754b7df69-sgsls" event={"ID":"bc51d97f-434c-4b55-b6ad-cb8e81c02db4","Type":"ContainerStarted","Data":"a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6"}
Oct 31 16:09:17.229797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:09:17.229748    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5754b7df69-sgsls" podStartSLOduration=1.797169301 podStartE2EDuration="3.229736426s" totalImagesPullingTime="1.432567125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:17.229407495 +0000 UTC m=+400.929336817" watchObservedRunningTime="2026-10-31 16:09:17.229736426 +0000 UTC m=+400.929665738"
Oct 31 16:09:24.002712 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:09:24.002660    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:09:54.002785 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:09:54.002749    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:10:24.002550 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:10:24.002513    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:10:49.512089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:49.512057    4519 generic.go:356] "Generic (PLEG): container finished" podID="71eb9ea0-765d-4684-b4ca-803942305b02" containerID="96df9e7119fd769fd0c8a068d67f0c974b6cd33f8219804429f16bf1b6dfe513" exitCode=1
Oct 31 16:10:49.512478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:49.512128    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc" event={"ID":"71eb9ea0-765d-4684-b4ca-803942305b02","Type":"ContainerDied","Data":"96df9e7119fd769fd0c8a068d67f0c974b6cd33f8219804429f16bf1b6dfe513"}
Oct 31 16:10:49.512478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:49.512444    4519 scope.go:122] "RemoveContainer" containerID="96df9e7119fd769fd0c8a068d67f0c974b6cd33f8219804429f16bf1b6dfe513"
Oct 31 16:10:50.517615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:50.517576    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc" event={"ID":"71eb9ea0-765d-4684-b4ca-803942305b02","Type":"ContainerStarted","Data":"fa0a011d8e7dfb7fbc85c95fe3c0a5974a33cb5252f3773cc9cf7cffc09d9447"}
Oct 31 16:10:50.518123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:50.517867    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:10:54.003907 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:10:54.003865    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:10:54.265497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:54.265402    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:10:58.585629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.585589    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"]
Oct 31 16:10:58.586208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.586184    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" containerName="authorino"
Oct 31 16:10:58.586260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.586214    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" containerName="authorino"
Oct 31 16:10:58.586355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.586344    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c049041-4cd3-400f-aaf0-1a7c1fc70a8c" containerName="authorino"
Oct 31 16:10:58.590101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.590079    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.642098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.642066    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"]
Oct 31 16:10:58.679650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.679608    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.679838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.679705    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9lx\" (UniqueName: \"kubernetes.io/projected/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-kube-api-access-7s9lx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.679838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.679751    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.679949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.679862    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.679949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.679940    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.680039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.680024    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.694299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.694272    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 16:10:58.694422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.694300    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 16:10:58.694422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.694326    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ngqjx\""
Oct 31 16:10:58.700187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.700137    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 16:10:58.781095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781060    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781104    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781222    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9lx\" (UniqueName: \"kubernetes.io/projected/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-kube-api-access-7s9lx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781259    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781342    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781387    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781622    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781728    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.781812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.781791    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.783423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.783392    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.783729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.783713    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.844222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.844153    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9lx\" (UniqueName: \"kubernetes.io/projected/7f00984f-32ff-4b00-ae35-5372d3fd4ac3-kube-api-access-7s9lx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2tnln\" (UID: \"7f00984f-32ff-4b00-ae35-5372d3fd4ac3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:58.906114 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:58.906083    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:10:59.308769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:59.308737    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"]
Oct 31 16:10:59.315525 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:10:59.315497    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f00984f_32ff_4b00_ae35_5372d3fd4ac3.slice/crio-b37522704c7e5a2c395e24adac1b4d6f3a944f2a4f74e1a3e81ed60dbd9a7a67 WatchSource:0}: Error finding container b37522704c7e5a2c395e24adac1b4d6f3a944f2a4f74e1a3e81ed60dbd9a7a67: Status 404 returned error can't find the container with id b37522704c7e5a2c395e24adac1b4d6f3a944f2a4f74e1a3e81ed60dbd9a7a67
Oct 31 16:10:59.560362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:10:59.560281    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln" event={"ID":"7f00984f-32ff-4b00-ae35-5372d3fd4ac3","Type":"ContainerStarted","Data":"b37522704c7e5a2c395e24adac1b4d6f3a944f2a4f74e1a3e81ed60dbd9a7a67"}
Oct 31 16:11:01.528303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:01.528278    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d6478c64c-rmrqc"
Oct 31 16:11:01.572543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:01.572506    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln" event={"ID":"7f00984f-32ff-4b00-ae35-5372d3fd4ac3","Type":"ContainerStarted","Data":"b54f432c88423a2c02e74f3fe11ad0a758e2b498904dc6c5a8ce65c8bdd1c03c"}
Oct 31 16:11:01.777064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:01.777009    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln" podStartSLOduration=2.19854186 podStartE2EDuration="3.776994635s" totalImagesPullingTime="1.578452775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:01.775994086 +0000 UTC m=+505.475923396" watchObservedRunningTime="2026-10-31 16:11:01.776994635 +0000 UTC m=+505.476923946"
Oct 31 16:11:01.907042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:01.906965    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:11:04.041515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:04.041480    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:11:07.017225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.017194    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-nxxwv"]
Oct 31 16:11:07.020977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.020958    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.070621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.070587    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 16:11:07.119086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.119059    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-nxxwv"]
Oct 31 16:11:07.176566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.176532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.176737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.176591    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-home\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.176737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.176642    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.176809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.176750    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26j7\" (UniqueName: \"kubernetes.io/projected/baf1235e-381e-406c-be7c-c19059168c8e-kube-api-access-r26j7\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.176809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.176786    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.176873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.176862    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf1235e-381e-406c-be7c-c19059168c8e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278093    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278182    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-home\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278208    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278281    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26j7\" (UniqueName: \"kubernetes.io/projected/baf1235e-381e-406c-be7c-c19059168c8e-kube-api-access-r26j7\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278313    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278348    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf1235e-381e-406c-be7c-c19059168c8e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278552    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278699    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-home\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.278764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.278743    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.280571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.280550    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf1235e-381e-406c-be7c-c19059168c8e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.280758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.280741    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf1235e-381e-406c-be7c-c19059168c8e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.332079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.332047    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26j7\" (UniqueName: \"kubernetes.io/projected/baf1235e-381e-406c-be7c-c19059168c8e-kube-api-access-r26j7\") pod \"echo-simulator-kserve-64b6f8f96c-nxxwv\" (UID: \"baf1235e-381e-406c-be7c-c19059168c8e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.332751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.332729    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:07.701706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:07.701609    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-nxxwv"]
Oct 31 16:11:07.708242 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:11:07.708214    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf1235e_381e_406c_be7c_c19059168c8e.slice/crio-7bd28c2954086f60e65560ca3e20d1cc253e3dc7b9e439384ccdacb0a81c3a3e WatchSource:0}: Error finding container 7bd28c2954086f60e65560ca3e20d1cc253e3dc7b9e439384ccdacb0a81c3a3e: Status 404 returned error can't find the container with id 7bd28c2954086f60e65560ca3e20d1cc253e3dc7b9e439384ccdacb0a81c3a3e
Oct 31 16:11:08.606529 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:08.606479    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv" event={"ID":"baf1235e-381e-406c-be7c-c19059168c8e","Type":"ContainerStarted","Data":"7bd28c2954086f60e65560ca3e20d1cc253e3dc7b9e439384ccdacb0a81c3a3e"}
Oct 31 16:11:08.906481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:08.906386    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:11:08.919548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:08.919519    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:11:09.622496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:09.622440    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln"
Oct 31 16:11:10.259570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.259539    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"]
Oct 31 16:11:10.263666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.263646    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.303539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.303511    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"]
Oct 31 16:11:10.313156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.313122    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 16:11:10.403811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.403775    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.404165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.403825    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.404165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.403945    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp26k\" (UniqueName: \"kubernetes.io/projected/602afcaa-d57d-4bde-aadb-646f36b8733f-kube-api-access-mp26k\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.404165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.404032    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.404165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.404085    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602afcaa-d57d-4bde-aadb-646f36b8733f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.404165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.404125    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505336    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505381    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505402    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505431    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp26k\" (UniqueName: \"kubernetes.io/projected/602afcaa-d57d-4bde-aadb-646f36b8733f-kube-api-access-mp26k\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505482    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505617    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602afcaa-d57d-4bde-aadb-646f36b8733f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505814    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.505888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.505873    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.507428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.507409    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602afcaa-d57d-4bde-aadb-646f36b8733f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.507807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.507789    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602afcaa-d57d-4bde-aadb-646f36b8733f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.560179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.560153    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp26k\" (UniqueName: \"kubernetes.io/projected/602afcaa-d57d-4bde-aadb-646f36b8733f-kube-api-access-mp26k\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx\" (UID: \"602afcaa-d57d-4bde-aadb-646f36b8733f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.580914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.580889    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:10.617622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.617585    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv" event={"ID":"baf1235e-381e-406c-be7c-c19059168c8e","Type":"ContainerStarted","Data":"dafd9be94ea1970aebd5f2b43069b92f4ab9536eb7fbd6747ae333b4cfc6c5fb"}
Oct 31 16:11:10.948398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.948286    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv" podStartSLOduration=2.178371744 podStartE2EDuration="4.948270038s" totalImagesPullingTime="2.769898294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:10.725442198 +0000 UTC m=+514.425371509" watchObservedRunningTime="2026-10-31 16:11:10.948270038 +0000 UTC m=+514.648199348"
Oct 31 16:11:10.949700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:10.949676    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"]
Oct 31 16:11:10.955368 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:11:10.955344    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602afcaa_d57d_4bde_aadb_646f36b8733f.slice/crio-1dbd9de6379f1b01eeb7e7e66741780d2f5a824083bb3bc8c5a82b8204723b94 WatchSource:0}: Error finding container 1dbd9de6379f1b01eeb7e7e66741780d2f5a824083bb3bc8c5a82b8204723b94: Status 404 returned error can't find the container with id 1dbd9de6379f1b01eeb7e7e66741780d2f5a824083bb3bc8c5a82b8204723b94
Oct 31 16:11:11.333442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:11.333413    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:11.623420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:11.623392    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx" event={"ID":"602afcaa-d57d-4bde-aadb-646f36b8733f","Type":"ContainerStarted","Data":"fa40f5bbda82762da1854e6a400eb59fbb02f238b9430b614369ea7c69705fb4"}
Oct 31 16:11:11.623543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:11.623444    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx" event={"ID":"602afcaa-d57d-4bde-aadb-646f36b8733f","Type":"ContainerStarted","Data":"1dbd9de6379f1b01eeb7e7e66741780d2f5a824083bb3bc8c5a82b8204723b94"}
Oct 31 16:11:11.730939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:11.730879    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx" podStartSLOduration=1.13206477 podStartE2EDuration="1.730861156s" totalImagesPullingTime="598.796386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:11.729928476 +0000 UTC m=+515.429857789" watchObservedRunningTime="2026-10-31 16:11:11.730861156 +0000 UTC m=+515.430790463"
Oct 31 16:11:12.581430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:12.581403    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:13.393079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.393045    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"]
Oct 31 16:11:13.396739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.396719    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.445996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.445959    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.445996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.446002    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.446253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.446037    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.446253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.446154    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac757df-4216-40c2-8219-c01aade2b9f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.446253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.446184    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.446253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.446217    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qbd\" (UniqueName: \"kubernetes.io/projected/8ac757df-4216-40c2-8219-c01aade2b9f3-kube-api-access-67qbd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.449563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.449535    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 16:11:13.470397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.470365    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"]
Oct 31 16:11:13.546941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.546910    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qbd\" (UniqueName: \"kubernetes.io/projected/8ac757df-4216-40c2-8219-c01aade2b9f3-kube-api-access-67qbd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547079    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547117    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547148    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547191    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac757df-4216-40c2-8219-c01aade2b9f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547610    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547629    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.547852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.547739    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.549398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.549377    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac757df-4216-40c2-8219-c01aade2b9f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.549578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.549560    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac757df-4216-40c2-8219-c01aade2b9f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.600988 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.600945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qbd\" (UniqueName: \"kubernetes.io/projected/8ac757df-4216-40c2-8219-c01aade2b9f3-kube-api-access-67qbd\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn\" (UID: \"8ac757df-4216-40c2-8219-c01aade2b9f3\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:13.711874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:13.711844    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:14.082622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:14.082521    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"]
Oct 31 16:11:14.112704 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:11:14.112600    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ac757df_4216_40c2_8219_c01aade2b9f3.slice/crio-670f6edba3c78bff112206f401aedd694d7fa4d23a7e6d4c3713fca87347cc25 WatchSource:0}: Error finding container 670f6edba3c78bff112206f401aedd694d7fa4d23a7e6d4c3713fca87347cc25: Status 404 returned error can't find the container with id 670f6edba3c78bff112206f401aedd694d7fa4d23a7e6d4c3713fca87347cc25
Oct 31 16:11:14.638404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:14.638368    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" event={"ID":"8ac757df-4216-40c2-8219-c01aade2b9f3","Type":"ContainerStarted","Data":"670f6edba3c78bff112206f401aedd694d7fa4d23a7e6d4c3713fca87347cc25"}
Oct 31 16:11:15.644473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:15.644417    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" event={"ID":"8ac757df-4216-40c2-8219-c01aade2b9f3","Type":"ContainerStarted","Data":"0d32f4ffe2a7446492397df9f51d0503b9eb9f97a4042d697646987fc475f897"}
Oct 31 16:11:15.712579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:15.712550    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:15.752937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:15.752891    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podStartSLOduration=2.1476464379999998 podStartE2EDuration="2.752878919s" totalImagesPullingTime="605.232481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:15.752503399 +0000 UTC m=+519.452432710" watchObservedRunningTime="2026-10-31 16:11:15.752878919 +0000 UTC m=+519.452808229"
Oct 31 16:11:17.333259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:17.333226    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:17.345785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:17.345759    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:17.663577 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:17.663506    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv"
Oct 31 16:11:20.581969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:20.581936    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:20.594187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:20.594153    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:20.677727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:20.677699    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx"
Oct 31 16:11:21.959388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:21.959352    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:11:23.712137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:23.712098    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:23.724687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:23.724657    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:24.004001 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:11:24.003922    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:11:24.694675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:24.694645    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:11:34.912512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:34.912481    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"]
Oct 31 16:11:34.918071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:34.918048    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:34.967576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:34.967546    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 16:11:35.043143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043114    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043143    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043164    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043222    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043302    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043354    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrbg\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043380    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043439    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.043581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.043472    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.072811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.072779    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gw56j\""
Oct 31 16:11:35.103596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.103562    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"]
Oct 31 16:11:35.144277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144248    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144279    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144299    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144420    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144466    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144514    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrbg\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144551    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144603    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144633    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144758    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144765    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.144995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.144968    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.145053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.145020    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.145359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.145340    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.146615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.146597    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.146937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.146916    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.198438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.198365    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.198588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.198506    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrbg\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4lhhw\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.230954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.230928    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:35.651096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.651063    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"]
Oct 31 16:11:35.698795 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:11:35.698762    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f4b5fd_e92c_4b54_9797_a3bafa886577.slice/crio-8de1b6b9f14759fbd6a70f2242e4d86d6c552dbfb310876b2f5e065045d202e2 WatchSource:0}: Error finding container 8de1b6b9f14759fbd6a70f2242e4d86d6c552dbfb310876b2f5e065045d202e2: Status 404 returned error can't find the container with id 8de1b6b9f14759fbd6a70f2242e4d86d6c552dbfb310876b2f5e065045d202e2
Oct 31 16:11:35.736158 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:35.736126    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" event={"ID":"58f4b5fd-e92c-4b54-9797-a3bafa886577","Type":"ContainerStarted","Data":"8de1b6b9f14759fbd6a70f2242e4d86d6c552dbfb310876b2f5e065045d202e2"}
Oct 31 16:11:39.907542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:39.907512    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:39.907815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:39.907578    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:39.907815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:39.907607    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.760816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:40.760780    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" event={"ID":"58f4b5fd-e92c-4b54-9797-a3bafa886577","Type":"ContainerStarted","Data":"90c934b0e67fc5b7511aa1e6eb5c2686b362b5bb376e2e8939c9871ecb53e0a8"}
Oct 31 16:11:40.979481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:40.973041    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podStartSLOduration=2.7670593610000003 podStartE2EDuration="6.973021132s" totalImagesPullingTime="4.205961771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:40.918935939 +0000 UTC m=+544.618865250" watchObservedRunningTime="2026-10-31 16:11:40.973021132 +0000 UTC m=+544.672950445"
Oct 31 16:11:41.231939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:41.231901    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:41.233312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:41.233286    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:41.233379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:41.233352    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:42.231581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:42.231544    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:42.231986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:42.231595    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:43.231923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:43.231890    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:43.232310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:43.231940    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:43.804935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:43.804900    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5754b7df69-sgsls"]
Oct 31 16:11:43.805161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:43.805132    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5754b7df69-sgsls" podUID="bc51d97f-434c-4b55-b6ad-cb8e81c02db4" containerName="authorino" containerID="cri-o://a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6" gracePeriod=30
Oct 31 16:11:43.857162 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:11:43.857134    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc51d97f_434c_4b55_b6ad_cb8e81c02db4.slice/crio-a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:11:44.103375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.103349    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:11:44.227596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.227564    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert\" (UniqueName: \"kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert\") pod \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") "
Oct 31 16:11:44.227596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.227602    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w\" (UniqueName: \"kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w\") pod \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\" (UID: \"bc51d97f-434c-4b55-b6ad-cb8e81c02db4\") "
Oct 31 16:11:44.229608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.229573    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w" pod "bc51d97f-434c-4b55-b6ad-cb8e81c02db4" (UID: "bc51d97f-434c-4b55-b6ad-cb8e81c02db4"). InnerVolumeSpecName "kube-api-access-4b72w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:44.231862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.231828    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:44.231987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.231879    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:44.237527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.237504    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert" pod "bc51d97f-434c-4b55-b6ad-cb8e81c02db4" (UID: "bc51d97f-434c-4b55-b6ad-cb8e81c02db4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:11:44.328511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.328416    4519 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-tls-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:11:44.328511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.328441    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b72w\" (UniqueName: \"kubernetes.io/projected/bc51d97f-434c-4b55-b6ad-cb8e81c02db4-kube-api-access-4b72w\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:11:44.805726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.805683    4519 generic.go:356] "Generic (PLEG): container finished" podID="bc51d97f-434c-4b55-b6ad-cb8e81c02db4" containerID="a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6" exitCode=0
Oct 31 16:11:44.805910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.805779    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5754b7df69-sgsls"
Oct 31 16:11:44.805910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.805780    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5754b7df69-sgsls" event={"ID":"bc51d97f-434c-4b55-b6ad-cb8e81c02db4","Type":"ContainerDied","Data":"a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6"}
Oct 31 16:11:44.805910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.805828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5754b7df69-sgsls" event={"ID":"bc51d97f-434c-4b55-b6ad-cb8e81c02db4","Type":"ContainerDied","Data":"ad297ab6333e388212ef8b18e5ba183f2d8f52d34dc566acb9d9e7a65abe0fdf"}
Oct 31 16:11:44.805910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.805846    4519 scope.go:122] "RemoveContainer" containerID="a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6"
Oct 31 16:11:44.814388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.814369    4519 scope.go:122] "RemoveContainer" containerID="a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6"
Oct 31 16:11:44.814674 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:11:44.814652    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6\": container with ID starting with a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6 not found: ID does not exist" containerID="a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6"
Oct 31 16:11:44.814737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.814681    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6"} err="failed to get container status \"a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6\": rpc error: code = NotFound desc = could not find container \"a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6\": container with ID starting with a08936ad4739b444faaae76aa77cabcbedd761db3d17e5a885cbf1ce9eb1d9a6 not found: ID does not exist"
Oct 31 16:11:44.960383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.960337    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5754b7df69-sgsls"]
Oct 31 16:11:44.961841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:44.961821    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5754b7df69-sgsls"]
Oct 31 16:11:45.232189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:45.232151    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:45.232644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:45.232234    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:46.231535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:46.231502    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:46.231749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:46.231561    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:46.439617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:46.439588    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc51d97f-434c-4b55-b6ad-cb8e81c02db4" path="/var/lib/kubelet/pods/bc51d97f-434c-4b55-b6ad-cb8e81c02db4/volumes"
Oct 31 16:11:47.231415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:47.231373    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4lhhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:11:47.231650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:47.231468    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 31 16:11:48.235868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:48.235838    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:48.236245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:48.236139    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:48.236840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:48.236821    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:11:50.653539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.653498    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"]
Oct 31 16:11:50.653925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.653881    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bc51d97f-434c-4b55-b6ad-cb8e81c02db4" containerName="authorino"
Oct 31 16:11:50.653925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.653892    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc51d97f-434c-4b55-b6ad-cb8e81c02db4" containerName="authorino"
Oct 31 16:11:50.653994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.653972    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc51d97f-434c-4b55-b6ad-cb8e81c02db4" containerName="authorino"
Oct 31 16:11:50.657108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.657090    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.709712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.709686    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"]
Oct 31 16:11:50.754470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.754436    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 16:11:50.754663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.754492    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b4pgc\""
Oct 31 16:11:50.789266 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.789224    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e25e0f-6faa-453c-970f-72924dbd21bf-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7bknp\" (UID: \"74e25e0f-6faa-453c-970f-72924dbd21bf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.789514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.789349    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/74e25e0f-6faa-453c-970f-72924dbd21bf-kube-api-access-7szpt\") pod \"payload-pre-processing-7b98bb678c-7bknp\" (UID: \"74e25e0f-6faa-453c-970f-72924dbd21bf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.889869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.889833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e25e0f-6faa-453c-970f-72924dbd21bf-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7bknp\" (UID: \"74e25e0f-6faa-453c-970f-72924dbd21bf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.890061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.889923    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/74e25e0f-6faa-453c-970f-72924dbd21bf-kube-api-access-7szpt\") pod \"payload-pre-processing-7b98bb678c-7bknp\" (UID: \"74e25e0f-6faa-453c-970f-72924dbd21bf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.890553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.890528    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e25e0f-6faa-453c-970f-72924dbd21bf-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7bknp\" (UID: \"74e25e0f-6faa-453c-970f-72924dbd21bf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.943433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.943354    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szpt\" (UniqueName: \"kubernetes.io/projected/74e25e0f-6faa-453c-970f-72924dbd21bf-kube-api-access-7szpt\") pod \"payload-pre-processing-7b98bb678c-7bknp\" (UID: \"74e25e0f-6faa-453c-970f-72924dbd21bf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:50.977755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:50.977721    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:51.353657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:51.353621    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"]
Oct 31 16:11:51.360431 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:11:51.360399    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e25e0f_6faa_453c_970f_72924dbd21bf.slice/crio-1cf136fd56d4a47b33f1dd47c5775f0a4f933aada0a11ffe540671df8d77628f WatchSource:0}: Error finding container 1cf136fd56d4a47b33f1dd47c5775f0a4f933aada0a11ffe540671df8d77628f: Status 404 returned error can't find the container with id 1cf136fd56d4a47b33f1dd47c5775f0a4f933aada0a11ffe540671df8d77628f
Oct 31 16:11:51.844577 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:51.844531    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp" event={"ID":"74e25e0f-6faa-453c-970f-72924dbd21bf","Type":"ContainerStarted","Data":"1cf136fd56d4a47b33f1dd47c5775f0a4f933aada0a11ffe540671df8d77628f"}
Oct 31 16:11:54.007031 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:11:54.006987    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:11:54.861964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:54.861912    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp" event={"ID":"74e25e0f-6faa-453c-970f-72924dbd21bf","Type":"ContainerStarted","Data":"d618fe4cdfff4b0cb9a09e5897202ecc5aacb49b817351a7fefdb035d666e588"}
Oct 31 16:11:54.862127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:54.861997    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:11:54.969981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:54.969928    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp" podStartSLOduration=2.024457657 podStartE2EDuration="4.969915096s" totalImagesPullingTime="2.945457439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:54.968443736 +0000 UTC m=+558.668373047" watchObservedRunningTime="2026-10-31 16:11:54.969915096 +0000 UTC m=+558.669844406"
Oct 31 16:11:59.589858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.589823    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"]
Oct 31 16:11:59.593316 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.593296    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.645575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.645539    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"]
Oct 31 16:11:59.682315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.682280    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74j5\" (UniqueName: \"kubernetes.io/projected/eadb7dc3-b107-489b-88f2-f3f69567833b-kube-api-access-z74j5\") pod \"ai-gateway-controller-79c59c6ccb-v6mpc\" (UID: \"eadb7dc3-b107-489b-88f2-f3f69567833b\") " pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.682494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.682382    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadb7dc3-b107-489b-88f2-f3f69567833b-metrics-certs\") pod \"ai-gateway-controller-79c59c6ccb-v6mpc\" (UID: \"eadb7dc3-b107-489b-88f2-f3f69567833b\") " pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.691117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.691093    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5s6jt\""
Oct 31 16:11:59.691117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.691115    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 16:11:59.691312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.691126    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 16:11:59.782858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.782821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadb7dc3-b107-489b-88f2-f3f69567833b-metrics-certs\") pod \"ai-gateway-controller-79c59c6ccb-v6mpc\" (UID: \"eadb7dc3-b107-489b-88f2-f3f69567833b\") " pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.783028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.782931    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74j5\" (UniqueName: \"kubernetes.io/projected/eadb7dc3-b107-489b-88f2-f3f69567833b-kube-api-access-z74j5\") pod \"ai-gateway-controller-79c59c6ccb-v6mpc\" (UID: \"eadb7dc3-b107-489b-88f2-f3f69567833b\") " pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.785352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.785324    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadb7dc3-b107-489b-88f2-f3f69567833b-metrics-certs\") pod \"ai-gateway-controller-79c59c6ccb-v6mpc\" (UID: \"eadb7dc3-b107-489b-88f2-f3f69567833b\") " pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.836888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.836858    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74j5\" (UniqueName: \"kubernetes.io/projected/eadb7dc3-b107-489b-88f2-f3f69567833b-kube-api-access-z74j5\") pod \"ai-gateway-controller-79c59c6ccb-v6mpc\" (UID: \"eadb7dc3-b107-489b-88f2-f3f69567833b\") " pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:11:59.908880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:11:59.908792    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:12:00.273774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:00.273741    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"]
Oct 31 16:12:00.280490 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:12:00.280446    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadb7dc3_b107_489b_88f2_f3f69567833b.slice/crio-0c14c75cedfd26b5d9d98b6f41887997a0d745caaaecc6fc4a6dad0d2be946ce WatchSource:0}: Error finding container 0c14c75cedfd26b5d9d98b6f41887997a0d745caaaecc6fc4a6dad0d2be946ce: Status 404 returned error can't find the container with id 0c14c75cedfd26b5d9d98b6f41887997a0d745caaaecc6fc4a6dad0d2be946ce
Oct 31 16:12:00.889405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:00.889372    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc" event={"ID":"eadb7dc3-b107-489b-88f2-f3f69567833b","Type":"ContainerStarted","Data":"0c14c75cedfd26b5d9d98b6f41887997a0d745caaaecc6fc4a6dad0d2be946ce"}
Oct 31 16:12:03.904876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:03.904839    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc" event={"ID":"eadb7dc3-b107-489b-88f2-f3f69567833b","Type":"ContainerStarted","Data":"13e3691bdfe191a786587ae2a30efb8af390e4eea96646d0721b9b187ee16704"}
Oct 31 16:12:03.905270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:03.905019    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:12:04.013535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:04.013488    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc" podStartSLOduration=2.032673175 podStartE2EDuration="5.013472727s" totalImagesPullingTime="2.980799552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:04.012173901 +0000 UTC m=+567.712103212" watchObservedRunningTime="2026-10-31 16:12:04.013472727 +0000 UTC m=+567.713402063"
Oct 31 16:12:05.869709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:05.869678    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7bknp"
Oct 31 16:12:14.911944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:14.911912    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79c59c6ccb-v6mpc"
Oct 31 16:12:19.913289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.913219    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"]
Oct 31 16:12:19.916788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.916766    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.963750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.963714    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.963904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.963762    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.963904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.963852    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.963904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.963889    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76758c23-3740-4e77-9f9e-577afcda8bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.964071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.963906    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mff8g\" (UniqueName: \"kubernetes.io/projected/76758c23-3740-4e77-9f9e-577afcda8bdf-kube-api-access-mff8g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.964071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.964058    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.964181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.964095    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.964181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.964131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:19.964259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:19.964183    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.015906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.015869    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"]
Oct 31 16:12:20.065204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065170    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065211    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76758c23-3740-4e77-9f9e-577afcda8bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065234    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mff8g\" (UniqueName: \"kubernetes.io/projected/76758c23-3740-4e77-9f9e-577afcda8bdf-kube-api-access-mff8g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065301    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065325    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065350    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065401    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065489    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065541    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065831 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065756    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065855    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.065971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.065950    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.066125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.066107    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.066179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.066130    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76758c23-3740-4e77-9f9e-577afcda8bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.067546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.067528    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.067996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.067975    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.119563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.119531    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76758c23-3740-4e77-9f9e-577afcda8bdf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.119675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.119650    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mff8g\" (UniqueName: \"kubernetes.io/projected/76758c23-3740-4e77-9f9e-577afcda8bdf-kube-api-access-mff8g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-d7kw8\" (UID: \"76758c23-3740-4e77-9f9e-577afcda8bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.229612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.229582    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:20.669244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.669159    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"]
Oct 31 16:12:20.680110 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.680066    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:20.680213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.680156    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:20.680213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.680199    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:20.982765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.982729    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" event={"ID":"76758c23-3740-4e77-9f9e-577afcda8bdf","Type":"ContainerStarted","Data":"e77322da4b7a9750e23783d9354f6fafde413e2bcddf25aacfe059bb54ff2699"}
Oct 31 16:12:20.983160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:20.982787    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" event={"ID":"76758c23-3740-4e77-9f9e-577afcda8bdf","Type":"ContainerStarted","Data":"4145800088b9afec3ca707854309093206ecf3b12a84036bcb5c408e410d94c1"}
Oct 31 16:12:21.095846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:21.095794    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podStartSLOduration=2.095779454 podStartE2EDuration="2.095779454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:21.094858769 +0000 UTC m=+584.794788082" watchObservedRunningTime="2026-10-31 16:12:21.095779454 +0000 UTC m=+584.795708765"
Oct 31 16:12:21.230088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:21.230024    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:21.231566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:21.231543    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:21.231697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:21.231583    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:22.230472 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:22.230433    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:22.230873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:22.230525    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:23.230335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:23.230292    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:23.230540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:23.230358    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:24.014306 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:12:24.014273    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:12:24.230858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:24.230820    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:24.231278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:24.230874    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:25.229970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:25.229934    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:25.230161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:25.229988    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:26.230723 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:26.230686    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:26.231128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:26.230741    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:27.230217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:27.230182    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-d7kw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 16:12:27.230392 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:27.230237    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8" podUID="76758c23-3740-4e77-9f9e-577afcda8bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 16:12:28.958469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:28.958422    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:28.958941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:28.958806    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:28.959676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:28.959655    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-d7kw8"
Oct 31 16:12:45.554602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.554565    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"]
Oct 31 16:12:45.571823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.571783    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.613683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.613648    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"]
Oct 31 16:12:45.658373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.658337    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a1685e-openshift-default-dockercfg-2cr54\""
Oct 31 16:12:45.725464 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725431    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725560    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725593    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725612    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725631    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725680    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bbt\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725756    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725796    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.725870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.725830    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827167    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bbt\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827200    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827217    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827253    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827316    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827380    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827413    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827440    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827499    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827662    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827734    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.827943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827932    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.828002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.827983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.828081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.828060    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.829793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.829768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.829922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.829902    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.882272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.882240    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bbt\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.882646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.882628    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:45.884620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:45.884602    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:46.317293 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.317260    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"]
Oct 31 16:12:46.325065 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:12:46.325036    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de72231_0262_40b5_87b4_8a2154737fa7.slice/crio-7cda9c63c60a2c8ee2d93aefa4ff4c5f0d4b0380ad7a5b6d2e71104c9b1ab2cc WatchSource:0}: Error finding container 7cda9c63c60a2c8ee2d93aefa4ff4c5f0d4b0380ad7a5b6d2e71104c9b1ab2cc: Status 404 returned error can't find the container with id 7cda9c63c60a2c8ee2d93aefa4ff4c5f0d4b0380ad7a5b6d2e71104c9b1ab2cc
Oct 31 16:12:46.328123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.328073    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.328237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.328175    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.328237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.328220    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.538349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.538320    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"]
Oct 31 16:12:46.548846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.548819    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.599420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.599349    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"]
Oct 31 16:12:46.605441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.605341    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-59357a-openshift-default-dockercfg-z7nvh\""
Oct 31 16:12:46.634790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.634759    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.634974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.634802    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.634974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.634837    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.634974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.634956    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.635161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.635005    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.635161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.635090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.635161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.635129    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.635314 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.635263    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.635314 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.635300    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4g2c\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736111    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736148    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736169    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736185    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736241    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4g2c\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736350    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736389    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736428    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736699    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736727    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.736921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736797    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.737020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.736979    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.737450 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.737431    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.738624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.738600    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.738839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.738823    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.790420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.790390    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.790616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.790597    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4g2c\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.832725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.832702    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"]
Oct 31 16:12:46.837394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.837377    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.882188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.879306    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:46.894286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.894257    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e7f13a-openshift-default-dockercfg-m7lj5\""
Oct 31 16:12:46.897651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.897628    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"]
Oct 31 16:12:46.937685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937654    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.937823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937695    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.937879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937820    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwq4f\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.937879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937854    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.937944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937896    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.937944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937918    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.938013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937953    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.938013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937972    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:46.938013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:46.937996    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039192    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039324    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039370    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039409    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039503    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039530    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039611    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwq4f\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039644    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039657    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.039944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.039918    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.040197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.040171    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.040278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.040195    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.041726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.041705    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.042122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.042103    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.093628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.093606    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.093931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.093906    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwq4f\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.114204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.114174    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" event={"ID":"0de72231-0262-40b5-87b4-8a2154737fa7","Type":"ContainerStarted","Data":"49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"}
Oct 31 16:12:47.114351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.114223    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" event={"ID":"0de72231-0262-40b5-87b4-8a2154737fa7","Type":"ContainerStarted","Data":"7cda9c63c60a2c8ee2d93aefa4ff4c5f0d4b0380ad7a5b6d2e71104c9b1ab2cc"}
Oct 31 16:12:47.155345 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.155280    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:47.227019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.226968    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podStartSLOduration=2.2269495089999998 podStartE2EDuration="2.226949509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:47.225686797 +0000 UTC m=+610.925616109" watchObservedRunningTime="2026-10-31 16:12:47.226949509 +0000 UTC m=+610.926878822"
Oct 31 16:12:47.274954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.274923    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"]
Oct 31 16:12:47.275700 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:12:47.275678    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2af1c77_80bd_4ed1_95cd_c35c8ef1080f.slice/crio-490f7182eacf4c528ed745a8d18cdf43dc4f7cb508a130f8fbaca53616b0ebf3 WatchSource:0}: Error finding container 490f7182eacf4c528ed745a8d18cdf43dc4f7cb508a130f8fbaca53616b0ebf3: Status 404 returned error can't find the container with id 490f7182eacf4c528ed745a8d18cdf43dc4f7cb508a130f8fbaca53616b0ebf3
Oct 31 16:12:47.278668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.278633    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.278751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.278714    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.278790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.278754    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.550138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.550107    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"]
Oct 31 16:12:47.555551 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:12:47.555526    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f39b36f_6f3f_4081_9e40_de9dbff3c623.slice/crio-3c245a093e9f464298cb32acbef319f8d3a03147fcbb4c344f10e24442e46058 WatchSource:0}: Error finding container 3c245a093e9f464298cb32acbef319f8d3a03147fcbb4c344f10e24442e46058: Status 404 returned error can't find the container with id 3c245a093e9f464298cb32acbef319f8d3a03147fcbb4c344f10e24442e46058
Oct 31 16:12:47.558019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.557978    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.558123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.558079    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.558164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.558126    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.885398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.885319    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:47.893574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:47.893546    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:48.121032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.120993    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" event={"ID":"6f39b36f-6f3f-4081-9e40-de9dbff3c623","Type":"ContainerStarted","Data":"0aeb0c1d0d2bb18fdf4990455db1fe789c3a97ff2e998b86cbe66b0dd7c75d19"}
Oct 31 16:12:48.121233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.121087    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" event={"ID":"6f39b36f-6f3f-4081-9e40-de9dbff3c623","Type":"ContainerStarted","Data":"3c245a093e9f464298cb32acbef319f8d3a03147fcbb4c344f10e24442e46058"}
Oct 31 16:12:48.122679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.122647    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" event={"ID":"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f","Type":"ContainerStarted","Data":"3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"}
Oct 31 16:12:48.122891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.122700    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" event={"ID":"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f","Type":"ContainerStarted","Data":"490f7182eacf4c528ed745a8d18cdf43dc4f7cb508a130f8fbaca53616b0ebf3"}
Oct 31 16:12:48.122891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.122843    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:48.123702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.123686    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:12:48.156097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.156017    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:48.161179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.161156    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:48.231361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.231304    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podStartSLOduration=2.231287195 podStartE2EDuration="2.231287195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:48.230670078 +0000 UTC m=+611.930599389" watchObservedRunningTime="2026-10-31 16:12:48.231287195 +0000 UTC m=+611.931216505"
Oct 31 16:12:48.342203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.342154    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podStartSLOduration=2.342140391 podStartE2EDuration="2.342140391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:48.340040166 +0000 UTC m=+612.039969478" watchObservedRunningTime="2026-10-31 16:12:48.342140391 +0000 UTC m=+612.042069702"
Oct 31 16:12:48.880429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.880401    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:48.885054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:48.885032    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:49.128543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:49.128084    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:49.128984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:49.128745    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:49.128984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:49.128807    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:12:49.128984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:49.128924    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:12:54.003448 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:12:54.003406    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:12:54.712208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.712177    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"]
Oct 31 16:12:54.715993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.715976    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:54.819491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.815314    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4fe3a4\""
Oct 31 16:12:54.819491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.815392    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4fe3a4-dockercfg-skqrh\""
Oct 31 16:12:54.819491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.819052    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"]
Oct 31 16:12:54.828427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.828398    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4lk\" (UniqueName: \"kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk\") pod \"payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:54.828596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.828543    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:54.884996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.884967    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"]
Oct 31 16:12:54.888939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.888912    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:54.929979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.929946    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4lk\" (UniqueName: \"kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk\") pod \"payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:54.930166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.930036    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:54.930827 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.930805    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:54.941596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.941571    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"]
Oct 31 16:12:54.984367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:54.984300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4lk\" (UniqueName: \"kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk\") pod \"payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:55.030135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.030105    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:55.030694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.030671    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.030845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.030824    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vfx\" (UniqueName: \"kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx\") pod \"payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.131888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.131855    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vfx\" (UniqueName: \"kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx\") pod \"payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.132083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.131918    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.132618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.132598    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.186792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.186762    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vfx\" (UniqueName: \"kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx\") pod \"payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.203753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.203730    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:55.399136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.399096    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"]
Oct 31 16:12:55.405193 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:12:55.405166    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce91a06d_620e_4c7f_9317_313db927a913.slice/crio-6549e9a4ba137a382a61bf1494c550f2c0647a02e87165403a66f06fd5847322 WatchSource:0}: Error finding container 6549e9a4ba137a382a61bf1494c550f2c0647a02e87165403a66f06fd5847322: Status 404 returned error can't find the container with id 6549e9a4ba137a382a61bf1494c550f2c0647a02e87165403a66f06fd5847322
Oct 31 16:12:55.570533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:55.570450    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"]
Oct 31 16:12:55.576109 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:12:55.576084    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba368270_cdbc_419e_85cb_74a6e8a0fab7.slice/crio-838a73725c962c7768c37320248cab9d25aa33e7259b1100090264088ceef530 WatchSource:0}: Error finding container 838a73725c962c7768c37320248cab9d25aa33e7259b1100090264088ceef530: Status 404 returned error can't find the container with id 838a73725c962c7768c37320248cab9d25aa33e7259b1100090264088ceef530
Oct 31 16:12:56.171317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.171275    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh" event={"ID":"ba368270-cdbc-419e-85cb-74a6e8a0fab7","Type":"ContainerStarted","Data":"ca6db2d2ac7503fe1740ad1b1e33d0eb7ae6af3cad61202546cc135e628bf330"}
Oct 31 16:12:56.171816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.171367    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh" event={"ID":"ba368270-cdbc-419e-85cb-74a6e8a0fab7","Type":"ContainerStarted","Data":"838a73725c962c7768c37320248cab9d25aa33e7259b1100090264088ceef530"}
Oct 31 16:12:56.171816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.171428    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:12:56.173047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.173016    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" event={"ID":"ce91a06d-620e-4c7f-9317-313db927a913","Type":"ContainerStarted","Data":"6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37"}
Oct 31 16:12:56.173171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.173053    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" event={"ID":"ce91a06d-620e-4c7f-9317-313db927a913","Type":"ContainerStarted","Data":"6549e9a4ba137a382a61bf1494c550f2c0647a02e87165403a66f06fd5847322"}
Oct 31 16:12:56.173171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.173073    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:12:56.282591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.282531    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh" podStartSLOduration=2.28251748 podStartE2EDuration="2.28251748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:56.280732399 +0000 UTC m=+619.980661714" watchObservedRunningTime="2026-10-31 16:12:56.28251748 +0000 UTC m=+619.982446806"
Oct 31 16:12:56.387894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:12:56.387844    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" podStartSLOduration=2.387827899 podStartE2EDuration="2.387827899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:56.3871144 +0000 UTC m=+620.087043712" watchObservedRunningTime="2026-10-31 16:12:56.387827899 +0000 UTC m=+620.087757213"
Oct 31 16:13:01.814578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.814542    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"]
Oct 31 16:13:01.818332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.818313    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:01.868000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.867960    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f7821a\""
Oct 31 16:13:01.895308 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.895280    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lc5\" (UniqueName: \"kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5\") pod \"payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:01.895497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.895364    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:01.915264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.915237    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f7821a-dockercfg-tmllb\""
Oct 31 16:13:01.917608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.917581    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"]
Oct 31 16:13:01.996081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.996053    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lc5\" (UniqueName: \"kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5\") pod \"payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:01.996250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.996122    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:01.997040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:01.997016    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:02.051856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:02.051825    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lc5\" (UniqueName: \"kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5\") pod \"payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:02.131910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:02.131828    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:02.514231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:02.514203    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"]
Oct 31 16:13:02.520904 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:13:02.520873    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585965f9_fd7f_4617_bac8_fa97dec765e1.slice/crio-81909a2b00e51eaa9337a7c7eb801616165d8d14509f2c0dcbe4afd386bb994b WatchSource:0}: Error finding container 81909a2b00e51eaa9337a7c7eb801616165d8d14509f2c0dcbe4afd386bb994b: Status 404 returned error can't find the container with id 81909a2b00e51eaa9337a7c7eb801616165d8d14509f2c0dcbe4afd386bb994b
Oct 31 16:13:03.210406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:03.210373    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2" event={"ID":"585965f9-fd7f-4617-bac8-fa97dec765e1","Type":"ContainerStarted","Data":"4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80"}
Oct 31 16:13:03.210802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:03.210436    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:03.210802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:03.210464    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2" event={"ID":"585965f9-fd7f-4617-bac8-fa97dec765e1","Type":"ContainerStarted","Data":"81909a2b00e51eaa9337a7c7eb801616165d8d14509f2c0dcbe4afd386bb994b"}
Oct 31 16:13:03.319383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:03.319330    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2" podStartSLOduration=2.319317867 podStartE2EDuration="2.319317867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:03.317209437 +0000 UTC m=+627.017138808" watchObservedRunningTime="2026-10-31 16:13:03.319317867 +0000 UTC m=+627.019247178"
Oct 31 16:13:04.806546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:04.806498    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:13:06.164205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.164169    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"]
Oct 31 16:13:06.173483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.168122    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.221420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.221392    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"]
Oct 31 16:13:06.242060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.242022    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svt76\" (UniqueName: \"kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76\") pod \"payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.242240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.242121    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.287480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.265661    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ac1273\""
Oct 31 16:13:06.290530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.289792    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ac1273-dockercfg-6p4f4\""
Oct 31 16:13:06.326990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.326957    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"]
Oct 31 16:13:06.330724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.330700    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.343216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.343185    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.343337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.343285    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svt76\" (UniqueName: \"kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76\") pod \"payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.344028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.344006    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.394655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.394625    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"]
Oct 31 16:13:06.402431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.402390    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svt76\" (UniqueName: \"kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76\") pod \"payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.444806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.444728    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.444806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.444781    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hhl\" (UniqueName: \"kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl\") pod \"payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.500738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.500707    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:06.546472 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.546431    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.546636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.546509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hhl\" (UniqueName: \"kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl\") pod \"payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.547317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.547290    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.600489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.600418    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hhl\" (UniqueName: \"kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl\") pod \"payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.644166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.644139    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:06.892395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:06.892360    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"]
Oct 31 16:13:06.894084 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:13:06.894048    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd236a06f_6b03_401c_97d6_1efae5cc2406.slice/crio-c5112c384e7e223729576afcabd4cad915223651a3e3fa0e2da78f6b1247ee8b WatchSource:0}: Error finding container c5112c384e7e223729576afcabd4cad915223651a3e3fa0e2da78f6b1247ee8b: Status 404 returned error can't find the container with id c5112c384e7e223729576afcabd4cad915223651a3e3fa0e2da78f6b1247ee8b
Oct 31 16:13:07.036291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.036216    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"]
Oct 31 16:13:07.042701 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:13:07.042676    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b3689d_353c_4ce9_99cd_f19df1dfceff.slice/crio-b3748dfc5f8029e04102430d9aa07068eddbfb977845734cdcedbd6ddc33def9 WatchSource:0}: Error finding container b3748dfc5f8029e04102430d9aa07068eddbfb977845734cdcedbd6ddc33def9: Status 404 returned error can't find the container with id b3748dfc5f8029e04102430d9aa07068eddbfb977845734cdcedbd6ddc33def9
Oct 31 16:13:07.191377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.191340    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:13:07.191905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.191885    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:13:07.249777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.249749    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" event={"ID":"d236a06f-6b03-401c-97d6-1efae5cc2406","Type":"ContainerStarted","Data":"8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769"}
Oct 31 16:13:07.249961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.249800    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" event={"ID":"d236a06f-6b03-401c-97d6-1efae5cc2406","Type":"ContainerStarted","Data":"c5112c384e7e223729576afcabd4cad915223651a3e3fa0e2da78f6b1247ee8b"}
Oct 31 16:13:07.249961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.249828    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:07.251225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.251203    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" event={"ID":"29b3689d-353c-4ce9-99cd-f19df1dfceff","Type":"ContainerStarted","Data":"8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b"}
Oct 31 16:13:07.251332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.251245    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:07.251332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.251260    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" event={"ID":"29b3689d-353c-4ce9-99cd-f19df1dfceff","Type":"ContainerStarted","Data":"b3748dfc5f8029e04102430d9aa07068eddbfb977845734cdcedbd6ddc33def9"}
Oct 31 16:13:07.413263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.413168    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" podStartSLOduration=1.413152616 podStartE2EDuration="1.413152616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:07.410450229 +0000 UTC m=+631.110379540" watchObservedRunningTime="2026-10-31 16:13:07.413152616 +0000 UTC m=+631.113081926"
Oct 31 16:13:07.517692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:07.517634    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" podStartSLOduration=1.517604167 podStartE2EDuration="1.517604167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:07.517089328 +0000 UTC m=+631.217018633" watchObservedRunningTime="2026-10-31 16:13:07.517604167 +0000 UTC m=+631.217533481"
Oct 31 16:13:10.763363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.763328    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"]
Oct 31 16:13:10.784563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.783008    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:10.819780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.819746    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"]
Oct 31 16:13:10.864033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.864006    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e7f13a\""
Oct 31 16:13:10.864033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.864019    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e7f13a-dockercfg-gjfnf\""
Oct 31 16:13:10.911882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.911835    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2r6\" (UniqueName: \"kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6\") pod \"payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:10.912090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.912013    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:10.913729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.913702    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"]
Oct 31 16:13:10.917447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.917425    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:10.970395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:10.970366    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"]
Oct 31 16:13:11.012919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.012891    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2r6\" (UniqueName: \"kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6\") pod \"payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:11.013054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.013011    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:11.013122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.013059    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrxh\" (UniqueName: \"kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh\") pod \"payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.013122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.013097    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.013833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.013790    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:11.066247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.066219    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2r6\" (UniqueName: \"kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6\") pod \"payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:11.101140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.101113    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:11.114800 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.114775    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrxh\" (UniqueName: \"kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh\") pod \"payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.114896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.114819    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.115746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.115722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.174691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.173590    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrxh\" (UniqueName: \"kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh\") pod \"payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.231765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.231730    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:11.472385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.472350    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"]
Oct 31 16:13:11.479227 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:13:11.479200    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013c6457_ed0f_4a16_959a_8bec9e9cec92.slice/crio-1614c1f1cbfb8b77c26bd6abd0bc885e10dd57c23bd72d50949805b2b457c008 WatchSource:0}: Error finding container 1614c1f1cbfb8b77c26bd6abd0bc885e10dd57c23bd72d50949805b2b457c008: Status 404 returned error can't find the container with id 1614c1f1cbfb8b77c26bd6abd0bc885e10dd57c23bd72d50949805b2b457c008
Oct 31 16:13:11.604089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:11.604063    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"]
Oct 31 16:13:11.610969 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:13:11.610945    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee7f68b_16e6_4338_b2ed_934506f988b4.slice/crio-96369b66e498dc4fc6e2deab8a937272048477c023fc78f4f63fd5c766f204f4 WatchSource:0}: Error finding container 96369b66e498dc4fc6e2deab8a937272048477c023fc78f4f63fd5c766f204f4: Status 404 returned error can't find the container with id 96369b66e498dc4fc6e2deab8a937272048477c023fc78f4f63fd5c766f204f4
Oct 31 16:13:12.279448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.279407    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" event={"ID":"013c6457-ed0f-4a16-959a-8bec9e9cec92","Type":"ContainerStarted","Data":"764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c"}
Oct 31 16:13:12.279901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.279524    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" event={"ID":"013c6457-ed0f-4a16-959a-8bec9e9cec92","Type":"ContainerStarted","Data":"1614c1f1cbfb8b77c26bd6abd0bc885e10dd57c23bd72d50949805b2b457c008"}
Oct 31 16:13:12.279901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.279559    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:12.281489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.281440    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" event={"ID":"6ee7f68b-16e6-4338-b2ed-934506f988b4","Type":"ContainerStarted","Data":"de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27"}
Oct 31 16:13:12.281649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.281510    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" event={"ID":"6ee7f68b-16e6-4338-b2ed-934506f988b4","Type":"ContainerStarted","Data":"96369b66e498dc4fc6e2deab8a937272048477c023fc78f4f63fd5c766f204f4"}
Oct 31 16:13:12.281649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.281531    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:12.388195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.388139    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" podStartSLOduration=2.388126272 podStartE2EDuration="2.388126272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:12.386520372 +0000 UTC m=+636.086449686" watchObservedRunningTime="2026-10-31 16:13:12.388126272 +0000 UTC m=+636.088055632"
Oct 31 16:13:12.495666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:12.495595    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" podStartSLOduration=2.495574918 podStartE2EDuration="2.495574918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:12.49404261 +0000 UTC m=+636.193971923" watchObservedRunningTime="2026-10-31 16:13:12.495574918 +0000 UTC m=+636.195504231"
Oct 31 16:13:13.979006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:13.978974    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:13:14.218911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:14.218883    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:13:15.302440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.302405    4519 generic.go:356] "Generic (PLEG): container finished" podID="0de72231-0262-40b5-87b4-8a2154737fa7" containerID="49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5" exitCode=0
Oct 31 16:13:15.302931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.302489    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" event={"ID":"0de72231-0262-40b5-87b4-8a2154737fa7","Type":"ContainerDied","Data":"49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"}
Oct 31 16:13:15.302931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.302912    4519 scope.go:122] "RemoveContainer" containerID="49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"
Oct 31 16:13:15.304591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.304553    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:15.304692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.304647    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:15.304731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.304693    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:15.582756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.582679    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"]
Oct 31 16:13:15.586620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.586600    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.636897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.636867    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ef6d93\""
Oct 31 16:13:15.636897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.636868    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ef6d93-dockercfg-2mptm\""
Oct 31 16:13:15.640303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.640277    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"]
Oct 31 16:13:15.668242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.668213    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.668407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.668253    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwgc\" (UniqueName: \"kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc\") pod \"payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.769623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.769593    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.769792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.769628    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwgc\" (UniqueName: \"kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc\") pod \"payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.770315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.770295    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.823444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.823418    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwgc\" (UniqueName: \"kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc\") pod \"payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:15.902038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:15.901973    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:16.275306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:16.273203    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"]
Oct 31 16:13:16.323282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:16.323243    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" event={"ID":"4a555145-7546-430a-90c6-a4a96b4a5106","Type":"ContainerStarted","Data":"21cfc8ecd2fdf2dc95b0f8285d6875f569a6d5b329dc9ad2356efbfb49c02d9b"}
Oct 31 16:13:16.326244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:16.326208    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" event={"ID":"0de72231-0262-40b5-87b4-8a2154737fa7","Type":"ContainerStarted","Data":"a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87"}
Oct 31 16:13:16.890689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:16.890647    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:13:16.891845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:16.891812    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 16:13:16.891964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:16.891858    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 16:13:17.333652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:17.333611    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" event={"ID":"4a555145-7546-430a-90c6-a4a96b4a5106","Type":"ContainerStarted","Data":"9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d"}
Oct 31 16:13:17.447881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:17.447830    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" podStartSLOduration=2.447818826 podStartE2EDuration="2.447818826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:17.44735151 +0000 UTC m=+641.147280822" watchObservedRunningTime="2026-10-31 16:13:17.447818826 +0000 UTC m=+641.147748137"
Oct 31 16:13:17.888074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:17.887603    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 16:13:17.888074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:17.887674    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 16:13:18.123903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:18.123855    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:13:18.258618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:18.258592    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:13:18.258779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:18.258665    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:13:18.338398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:18.338366    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:18.889590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:18.889551    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 16:13:18.889876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:18.889846    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 16:13:19.887479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:19.886832    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 16:13:19.887479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:19.886895    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 16:13:20.890940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:20.890897    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 16:13:20.891330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:20.890953    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 16:13:21.891144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:21.891113    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:13:21.892237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:21.892216    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:13:23.291754 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:23.291726    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:13:23.292135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:23.291791    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:13:24.003405 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:13:24.003363    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:13:29.346872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:29.346840    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:13:31.639111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.639079    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"]
Oct 31 16:13:31.644122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.644100    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.697969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.697929    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"]
Oct 31 16:13:31.741653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.741623    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ac1273-openshift-default-dockercfg-zfcwg\""
Oct 31 16:13:31.752111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752079    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752210    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59h5b\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752284    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752314    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752546    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752588    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.752637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.752625    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853129    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853245    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853276    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853318    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853400    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853429    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59h5b\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853488    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853527    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853613    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853800    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.853978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.853950    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.854055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.854035    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.854286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.854269    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.855672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.855648    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.855907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.855888    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.908591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.908523    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.908716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.908665    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59h5b\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:31.957074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:31.957036    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:32.276717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:32.276681    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:13:32.443495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:32.443446    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"]
Oct 31 16:13:32.449570 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:13:32.449535    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c0badf_2977_4f9a_8b8b_37d295c3f1a9.slice/crio-98562429c75f007b5b9118638da9d1d85af905b5dcb82fb1a83f7bf5c824fe2e WatchSource:0}: Error finding container 98562429c75f007b5b9118638da9d1d85af905b5dcb82fb1a83f7bf5c824fe2e: Status 404 returned error can't find the container with id 98562429c75f007b5b9118638da9d1d85af905b5dcb82fb1a83f7bf5c824fe2e
Oct 31 16:13:32.451826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:32.451793    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:32.451917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:32.451858    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:32.451917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:32.451886    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:33.422116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:33.422065    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" event={"ID":"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9","Type":"ContainerStarted","Data":"7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d"}
Oct 31 16:13:33.422604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:33.422177    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" event={"ID":"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9","Type":"ContainerStarted","Data":"98562429c75f007b5b9118638da9d1d85af905b5dcb82fb1a83f7bf5c824fe2e"}
Oct 31 16:13:33.550995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:33.550948    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podStartSLOduration=2.550935654 podStartE2EDuration="2.550935654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:33.550138244 +0000 UTC m=+657.250067555" watchObservedRunningTime="2026-10-31 16:13:33.550935654 +0000 UTC m=+657.250864959"
Oct 31 16:13:33.957731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:33.957695    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:33.959393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:33.959361    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 16:13:33.959556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:33.959417    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 16:13:34.958439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:34.958401    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 16:13:34.958863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:34.958491    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 16:13:35.957442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:35.957405    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 16:13:35.957672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:35.957498    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 16:13:36.958066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:36.958027    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 16:13:36.958557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:36.958104    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 16:13:37.957414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:37.957379    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 31 16:13:37.957610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:37.957439    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 31 16:13:38.961665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:38.961637    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:38.962041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:38.961900    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:38.962680 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:38.962664    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:13:40.364748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:40.364715    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:13:49.574162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:49.573692    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:42014->10.134.0.41:15021: read: connection reset by peer" start-of-body=
Oct 31 16:13:49.574162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:49.573753    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:42014->10.134.0.41:15021: read: connection reset by peer"
Oct 31 16:13:50.101858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.101072    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:58434->10.134.0.42:15021: read: connection reset by peer" start-of-body=
Oct 31 16:13:50.101858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.101161    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:58434->10.134.0.42:15021: read: connection reset by peer"
Oct 31 16:13:50.562491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.562435    4519 generic.go:356] "Generic (PLEG): container finished" podID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerID="0aeb0c1d0d2bb18fdf4990455db1fe789c3a97ff2e998b86cbe66b0dd7c75d19" exitCode=0
Oct 31 16:13:50.562674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.562510    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" event={"ID":"6f39b36f-6f3f-4081-9e40-de9dbff3c623","Type":"ContainerDied","Data":"0aeb0c1d0d2bb18fdf4990455db1fe789c3a97ff2e998b86cbe66b0dd7c75d19"}
Oct 31 16:13:50.562964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.562943    4519 scope.go:122] "RemoveContainer" containerID="0aeb0c1d0d2bb18fdf4990455db1fe789c3a97ff2e998b86cbe66b0dd7c75d19"
Oct 31 16:13:50.564397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.564362    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:50.564519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.564399    4519 generic.go:356] "Generic (PLEG): container finished" podID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerID="3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff" exitCode=0
Oct 31 16:13:50.564519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.564467    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:50.564519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.564476    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" event={"ID":"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f","Type":"ContainerDied","Data":"3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"}
Oct 31 16:13:50.564691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.564519    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:50.564782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.564767    4519 scope.go:122] "RemoveContainer" containerID="3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"
Oct 31 16:13:50.565912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.565888    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:50.565978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.565944    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:50.566016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:50.565975    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:51.601480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:51.601266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" event={"ID":"6f39b36f-6f3f-4081-9e40-de9dbff3c623","Type":"ContainerStarted","Data":"9d6bddee6e568fd63b6dfd8f35cbb20e3984f84c15592b4a06c3a4deab3e3c19"}
Oct 31 16:13:51.617274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:51.617234    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" event={"ID":"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f","Type":"ContainerStarted","Data":"d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f"}
Oct 31 16:13:51.883943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:51.883852    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:13:51.884307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:51.884216    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 16:13:51.884495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:51.884446    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 16:13:52.155725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:52.155647    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:13:52.155877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:52.155821    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 16:13:52.155921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:52.155873    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 16:13:52.881919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:52.881886    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 16:13:52.882566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:52.882526    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 16:13:53.155953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:53.155874    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 16:13:53.155953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:53.155921    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 16:13:53.883480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:53.881867    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 16:13:53.883480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:53.881936    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 16:13:54.063756 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:13:54.063701    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:13:54.155963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:54.155871    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 16:13:54.155963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:54.155927    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 16:13:54.712475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:54.712420    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:13:54.887478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:54.886850    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 16:13:54.887478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:54.886915    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 16:13:55.156504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:55.156391    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 16:13:55.156504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:55.156445    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 16:13:55.887487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:55.886884    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 16:13:55.887487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:55.886943    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 16:13:56.156619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:56.156520    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 31 16:13:56.156619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:56.156575    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 31 16:13:56.883421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:56.883394    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:13:56.883799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:56.883781    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:13:56.884532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:56.884510    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:13:57.160278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:57.160201    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:13:57.160769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:57.160603    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:13:57.161360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:13:57.161339    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:14:01.874368 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.874333    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"]
Oct 31 16:14:01.879402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.879383    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971225    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971347    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971430    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971519    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971548    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971565    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971597    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6nc\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.971803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.971646    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:01.978628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:01.978603    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"]
Oct 31 16:14:02.072765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.072731    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.072977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.072775    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.072969    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073036    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073084    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6nc\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073111    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073206    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073216    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073253    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073297    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073533    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073608    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.073938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.073921    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.075119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.075098    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.075400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.075380    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.129256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.129163    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.129533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.129272    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6nc\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.192384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.192351    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:02.673646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.673614    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"]
Oct 31 16:14:02.680381 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:14:02.680345    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1d4f62_fb25_460a_a12c_02ff0065a5f7.slice/crio-875e06f36d70868d4ba2c4694ac78dc4c9041d5013bf03e06d9e4d3dd6ec49a7 WatchSource:0}: Error finding container 875e06f36d70868d4ba2c4694ac78dc4c9041d5013bf03e06d9e4d3dd6ec49a7: Status 404 returned error can't find the container with id 875e06f36d70868d4ba2c4694ac78dc4c9041d5013bf03e06d9e4d3dd6ec49a7
Oct 31 16:14:02.682645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.682612    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:14:02.682752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.682680    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:14:02.682752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:02.682718    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:14:03.702014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:03.701961    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" event={"ID":"ef1d4f62-fb25-460a-a12c-02ff0065a5f7","Type":"ContainerStarted","Data":"b139a62818aa0df6503c1be5dce56bcb2a0607ce00776c5b70d42a90ad631802"}
Oct 31 16:14:03.702544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:03.702076    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" event={"ID":"ef1d4f62-fb25-460a-a12c-02ff0065a5f7","Type":"ContainerStarted","Data":"875e06f36d70868d4ba2c4694ac78dc4c9041d5013bf03e06d9e4d3dd6ec49a7"}
Oct 31 16:14:03.864436 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:03.864382    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podStartSLOduration=2.864365952 podStartE2EDuration="2.864365952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:03.862885213 +0000 UTC m=+687.562814538" watchObservedRunningTime="2026-10-31 16:14:03.864365952 +0000 UTC m=+687.564295263"
Oct 31 16:14:04.195523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:04.195476    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:04.196291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:04.196263    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:14:04.196423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:04.196322    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 16:14:05.192963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:05.192926    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:14:05.193467 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:05.192979    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 16:14:06.192980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:06.192940    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:14:06.193400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:06.193009    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 16:14:07.193370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:07.193337    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:14:07.193806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:07.193401    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 16:14:08.193898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:08.193857    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:14:08.194579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:08.194538    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 16:14:09.197149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:09.197113    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:09.197536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:09.197414    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:09.198093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:09.198066    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:14:22.208313 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:22.208274    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:14:24.004048 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:14:24.004000    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:14:31.608269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.608235    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"]
Oct 31 16:14:31.612302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.612283    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.685445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.685413    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"]
Oct 31 16:14:31.706851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.706814    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wpg\" (UniqueName: \"kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.706993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.706852    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.706993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.706918    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.707142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.707000    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.707142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.707053    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.707204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.707151    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.712516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.712492    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"default-dockercfg-4f4mn\""
Oct 31 16:14:31.712609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.712529    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"openshift-service-ca.crt\""
Oct 31 16:14:31.712609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.712537    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"kube-root-ca.crt\""
Oct 31 16:14:31.712609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.712529    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"simulator-w0-ac1273-kserve-self-signed-certs\""
Oct 31 16:14:31.808477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.808430    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.808663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.808549    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.808737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.808662    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wpg\" (UniqueName: \"kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.808737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.808697    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.808737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.808723    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.808905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.808761    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.809049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.809024    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.809117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.809035    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.809155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.809111    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.810817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.810795    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.811074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.811060    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.867524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.864974    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wpg\" (UniqueName: \"kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg\") pod \"simulator-w0-ac1273-kserve-8c5d4d776-c7jjn\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") " pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:31.925329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:31.925290    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:32.294213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:32.294171    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"]
Oct 31 16:14:32.300036 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:14:32.300010    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6f8a716_8d61_4c9e_af7f_0e1c0bd1572a.slice/crio-5f3a2c54f416c7a4984221db9fb0b66cf1abb2abb1d5c771426b49804c90d227 WatchSource:0}: Error finding container 5f3a2c54f416c7a4984221db9fb0b66cf1abb2abb1d5c771426b49804c90d227: Status 404 returned error can't find the container with id 5f3a2c54f416c7a4984221db9fb0b66cf1abb2abb1d5c771426b49804c90d227
Oct 31 16:14:32.871211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:32.871179    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" event={"ID":"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a","Type":"ContainerStarted","Data":"6dfd4818f996f225c832dfc2d74b156ac6770734a23c95ef47a180b78c183de7"}
Oct 31 16:14:32.871624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:32.871220    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" event={"ID":"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a","Type":"ContainerStarted","Data":"5f3a2c54f416c7a4984221db9fb0b66cf1abb2abb1d5c771426b49804c90d227"}
Oct 31 16:14:32.925549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:32.925521    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:32.979462 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:32.979416    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podStartSLOduration=1.979403048 podStartE2EDuration="1.979403048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:32.978869248 +0000 UTC m=+716.678798562" watchObservedRunningTime="2026-10-31 16:14:32.979403048 +0000 UTC m=+716.679332348"
Oct 31 16:14:36.705331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:36.705299    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:14:41.925442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:41.925405    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:41.937945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:41.937920    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:41.948868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:41.948849    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:14:54.006519 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:14:54.006482    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:14:57.633739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.633709    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"]
Oct 31 16:14:57.637780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.637764    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.690606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.690571    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"]
Oct 31 16:14:57.734766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.734719    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"default-dockercfg-7hqvj\""
Oct 31 16:14:57.734766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.734738    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"openshift-service-ca.crt\""
Oct 31 16:14:57.734973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.734793    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"kube-root-ca.crt\""
Oct 31 16:14:57.738073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.738056    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"premium-w6-a1685e-kserve-self-signed-certs\""
Oct 31 16:14:57.778724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.778692    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.782151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.778739    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.782151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.778870    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.782151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.778984    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.782151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.779018    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.782151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.779035    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktdnn\" (UniqueName: \"kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.879767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.879729    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.879945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.879812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.879945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.879830    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.879945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.879847    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktdnn\" (UniqueName: \"kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.879945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.879902    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.880157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.880020    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.880201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.880170    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.880243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.880223    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.880386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.880369    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.882208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.882187    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.882313 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.882298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.933939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.933870    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktdnn\" (UniqueName: \"kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn\") pod \"premium-w6-a1685e-kserve-8688dcc666-c4nvf\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") " pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:57.950200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:57.950172    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:14:58.338439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:58.338402    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"]
Oct 31 16:14:58.345678 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:14:58.345650    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15bdcd3c_9b39_4953_a2d4_651157576b56.slice/crio-9d8821b5b214dbb1c617f09f9980e0dadca366e0b08cad51406d3aeca47b3bdc WatchSource:0}: Error finding container 9d8821b5b214dbb1c617f09f9980e0dadca366e0b08cad51406d3aeca47b3bdc: Status 404 returned error can't find the container with id 9d8821b5b214dbb1c617f09f9980e0dadca366e0b08cad51406d3aeca47b3bdc
Oct 31 16:14:59.036152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:59.036113    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" event={"ID":"15bdcd3c-9b39-4953-a2d4-651157576b56","Type":"ContainerStarted","Data":"9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400"}
Oct 31 16:14:59.036601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:59.036163    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" event={"ID":"15bdcd3c-9b39-4953-a2d4-651157576b56","Type":"ContainerStarted","Data":"9d8821b5b214dbb1c617f09f9980e0dadca366e0b08cad51406d3aeca47b3bdc"}
Oct 31 16:14:59.146602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:59.146548    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podStartSLOduration=2.146529246 podStartE2EDuration="2.146529246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:59.144117708 +0000 UTC m=+742.844047019" watchObservedRunningTime="2026-10-31 16:14:59.146529246 +0000 UTC m=+742.846458558"
Oct 31 16:14:59.950614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:14:59.950577    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:15:00.165182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.165149    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"]
Oct 31 16:15:00.169952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.169927    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"]
Oct 31 16:15:00.170097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.170029    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.174283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.174261    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"]
Oct 31 16:15:00.174387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.174353    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.178044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.178024    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.212793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.212770    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"]
Oct 31 16:15:00.216770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.216736    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"]
Oct 31 16:15:00.216899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.216857    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.219995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.219972    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:15:00.219995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.219987    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:15:00.220180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.220058    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:15:00.220592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.220573    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.259899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.259872    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"]
Oct 31 16:15:00.262772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.262751    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"]
Oct 31 16:15:00.301436 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.301393    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"]
Oct 31 16:15:00.304569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.304541    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"]
Oct 31 16:15:00.308301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308277    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"]
Oct 31 16:15:00.308428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308402    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.308539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308487    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znd6\" (UniqueName: \"kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.308630 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308606    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhbt\" (UniqueName: \"kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt\") pod \"maas-api-key-cleanup-29891055-q8h2v\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.308748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308705    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gf8r\" (UniqueName: \"kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.308748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308731    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cks7t\" (UniqueName: \"kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.308874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308804    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.308874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308849    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.308989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308876    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkj4\" (UniqueName: \"kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.308989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308913    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891055-q8h2v\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.308989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.308954    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.409984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.409944    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhbt\" (UniqueName: \"kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt\") pod \"maas-api-key-cleanup-29891055-q8h2v\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.410160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410057    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf8r\" (UniqueName: \"kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.410160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410095    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cks7t\" (UniqueName: \"kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.410245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410163    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.410328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410304    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.410394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410374    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkj4\" (UniqueName: \"kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.410465 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410426    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891055-q8h2v\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.410524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410480    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.410578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.410628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.410612    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znd6\" (UniqueName: \"kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.411103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.411081    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.411184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.411099    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.411270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.411248    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891055-q8h2v\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.411302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.411249    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.411339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.411312    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.464199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.464123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cks7t\" (UniqueName: \"kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.464386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.464363    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhbt\" (UniqueName: \"kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt\") pod \"maas-api-key-cleanup-29891055-q8h2v\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.464478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.464363    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gf8r\" (UniqueName: \"kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.485094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.485063    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:00.490570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.490546    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkj4\" (UniqueName: \"kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.490673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.490656    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znd6\" (UniqueName: \"kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.493316 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.493291    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:00.499155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.499130    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:00.532354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.532314    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:00.542880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.538574    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:00.887300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.887232    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"]
Oct 31 16:15:00.894107 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:00.894073    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42942174_e286_4d8b_bf4b_dc8f07371e13.slice/crio-7c0fd6b551d162906b9dc611127cb1b4d3ef141940fa632ae5d2e44d6efabab5 WatchSource:0}: Error finding container 7c0fd6b551d162906b9dc611127cb1b4d3ef141940fa632ae5d2e44d6efabab5: Status 404 returned error can't find the container with id 7c0fd6b551d162906b9dc611127cb1b4d3ef141940fa632ae5d2e44d6efabab5
Oct 31 16:15:00.896656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.896639    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:15:00.939890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.939862    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"]
Oct 31 16:15:00.942465 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.942425    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"]
Oct 31 16:15:00.946600 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:00.946575    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2570659_f361_408c_b860_fba8eee4763b.slice/crio-6260c162bd0fdd9a9511945f47e104bdf6707db8bbaa16b73cc2269d2132c1c6 WatchSource:0}: Error finding container 6260c162bd0fdd9a9511945f47e104bdf6707db8bbaa16b73cc2269d2132c1c6: Status 404 returned error can't find the container with id 6260c162bd0fdd9a9511945f47e104bdf6707db8bbaa16b73cc2269d2132c1c6
Oct 31 16:15:00.948423 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:00.948402    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235ee35f_9f11_48fc_a583_1fb22e1e4327.slice/crio-4fe8213b1c0a14bbf7d97321f6860aa9a2b30877f407b4d948cfdf9c1779edcb WatchSource:0}: Error finding container 4fe8213b1c0a14bbf7d97321f6860aa9a2b30877f407b4d948cfdf9c1779edcb: Status 404 returned error can't find the container with id 4fe8213b1c0a14bbf7d97321f6860aa9a2b30877f407b4d948cfdf9c1779edcb
Oct 31 16:15:00.994215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.994180    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"]
Oct 31 16:15:00.997923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:00.997889    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"]
Oct 31 16:15:01.000914 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:01.000883    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41d42c9_376a_4d61_88b3_a01bb16b31f4.slice/crio-0cec89922cb552b18b1c5bfb5bc5fedcaef89756314dcc42419d2ed4264c8dae WatchSource:0}: Error finding container 0cec89922cb552b18b1c5bfb5bc5fedcaef89756314dcc42419d2ed4264c8dae: Status 404 returned error can't find the container with id 0cec89922cb552b18b1c5bfb5bc5fedcaef89756314dcc42419d2ed4264c8dae
Oct 31 16:15:01.002422 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:01.002393    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e1f774_b10a_4ad8_9f70_cf79ce1669ab.slice/crio-308fd237b2c7fb4bb115a2815e452551c3c4d4b3b1eaa5bd2984c87c694f8e6f WatchSource:0}: Error finding container 308fd237b2c7fb4bb115a2815e452551c3c4d4b3b1eaa5bd2984c87c694f8e6f: Status 404 returned error can't find the container with id 308fd237b2c7fb4bb115a2815e452551c3c4d4b3b1eaa5bd2984c87c694f8e6f
Oct 31 16:15:01.048738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:01.048703    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerStarted","Data":"7c0fd6b551d162906b9dc611127cb1b4d3ef141940fa632ae5d2e44d6efabab5"}
Oct 31 16:15:01.049898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:01.049874    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerStarted","Data":"4fe8213b1c0a14bbf7d97321f6860aa9a2b30877f407b4d948cfdf9c1779edcb"}
Oct 31 16:15:01.051060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:01.051040    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerStarted","Data":"0cec89922cb552b18b1c5bfb5bc5fedcaef89756314dcc42419d2ed4264c8dae"}
Oct 31 16:15:01.052176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:01.052146    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerStarted","Data":"308fd237b2c7fb4bb115a2815e452551c3c4d4b3b1eaa5bd2984c87c694f8e6f"}
Oct 31 16:15:01.053381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:01.053359    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerStarted","Data":"6260c162bd0fdd9a9511945f47e104bdf6707db8bbaa16b73cc2269d2132c1c6"}
Oct 31 16:15:03.187195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:03.187159    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:15:07.101597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.101558    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerStarted","Data":"e149d10d744aa0c7cc85b2684ff11cdcd5c4f72d714e69536bea2b5e52db4961"}
Oct 31 16:15:07.103237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.103212    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerStarted","Data":"740832e9b364b30379e2e87dadf2a9e445c3e0feb4dc263d4c98501cbf3d4e1a"}
Oct 31 16:15:07.104797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.104776    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerStarted","Data":"83f84464f4d02fb774c322c4d4a2e99c237312207fd82eac9ce0af0d289e69bc"}
Oct 31 16:15:07.106425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.106394    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerStarted","Data":"b48e01f506c3b98cde6ff27b4c09bc9c99ed41ebc9788378b94509c51ea28b03"}
Oct 31 16:15:07.107792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.107773    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerStarted","Data":"02e54e87e76080f1dde0b2996dd7b9bea491b7a062177a73ed082b998c9415bc"}
Oct 31 16:15:07.209125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.209075    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" podStartSLOduration=1.573373267 podStartE2EDuration="7.209060989s" totalImagesPullingTime="5.635687722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:07.207573084 +0000 UTC m=+750.907502399" watchObservedRunningTime="2026-10-31 16:15:07.209060989 +0000 UTC m=+750.908990322"
Oct 31 16:15:07.313997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.313944    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" podStartSLOduration=1.8646562169999998 podStartE2EDuration="7.313927423s" totalImagesPullingTime="5.449271206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:07.313537289 +0000 UTC m=+751.013466600" watchObservedRunningTime="2026-10-31 16:15:07.313927423 +0000 UTC m=+751.013856734"
Oct 31 16:15:07.422890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.422842    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" podStartSLOduration=1.9849428759999999 podStartE2EDuration="7.422828898s" totalImagesPullingTime="5.437886022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:07.420379194 +0000 UTC m=+751.120308506" watchObservedRunningTime="2026-10-31 16:15:07.422828898 +0000 UTC m=+751.122758241"
Oct 31 16:15:07.528927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.528827    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" podStartSLOduration=2.017122727 podStartE2EDuration="7.528813474s" totalImagesPullingTime="5.511690747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:07.526700803 +0000 UTC m=+751.226630115" watchObservedRunningTime="2026-10-31 16:15:07.528813474 +0000 UTC m=+751.228742786"
Oct 31 16:15:07.635010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.634956    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" podStartSLOduration=2.121418211 podStartE2EDuration="7.634943775s" totalImagesPullingTime="5.513525564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:07.632417621 +0000 UTC m=+751.332346931" watchObservedRunningTime="2026-10-31 16:15:07.634943775 +0000 UTC m=+751.334873085"
Oct 31 16:15:07.951168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.951069    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:15:07.963757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:07.963724    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:15:08.115255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.115226    4519 generic.go:356] "Generic (PLEG): container finished" podID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerID="83f84464f4d02fb774c322c4d4a2e99c237312207fd82eac9ce0af0d289e69bc" exitCode=60
Oct 31 16:15:08.115715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.115311    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerDied","Data":"83f84464f4d02fb774c322c4d4a2e99c237312207fd82eac9ce0af0d289e69bc"}
Oct 31 16:15:08.115715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.115565    4519 scope.go:122] "RemoveContainer" containerID="83f84464f4d02fb774c322c4d4a2e99c237312207fd82eac9ce0af0d289e69bc"
Oct 31 16:15:08.116990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.116873    4519 generic.go:356] "Generic (PLEG): container finished" podID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerID="b48e01f506c3b98cde6ff27b4c09bc9c99ed41ebc9788378b94509c51ea28b03" exitCode=60
Oct 31 16:15:08.117050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.116996    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerDied","Data":"b48e01f506c3b98cde6ff27b4c09bc9c99ed41ebc9788378b94509c51ea28b03"}
Oct 31 16:15:08.117165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.117149    4519 scope.go:122] "RemoveContainer" containerID="b48e01f506c3b98cde6ff27b4c09bc9c99ed41ebc9788378b94509c51ea28b03"
Oct 31 16:15:08.118711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.118685    4519 generic.go:356] "Generic (PLEG): container finished" podID="d2570659-f361-408c-b860-fba8eee4763b" containerID="02e54e87e76080f1dde0b2996dd7b9bea491b7a062177a73ed082b998c9415bc" exitCode=60
Oct 31 16:15:08.118790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.118712    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerDied","Data":"02e54e87e76080f1dde0b2996dd7b9bea491b7a062177a73ed082b998c9415bc"}
Oct 31 16:15:08.118961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.118940    4519 scope.go:122] "RemoveContainer" containerID="02e54e87e76080f1dde0b2996dd7b9bea491b7a062177a73ed082b998c9415bc"
Oct 31 16:15:08.120418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.120400    4519 generic.go:356] "Generic (PLEG): container finished" podID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerID="e149d10d744aa0c7cc85b2684ff11cdcd5c4f72d714e69536bea2b5e52db4961" exitCode=60
Oct 31 16:15:08.120528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.120503    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerDied","Data":"e149d10d744aa0c7cc85b2684ff11cdcd5c4f72d714e69536bea2b5e52db4961"}
Oct 31 16:15:08.120668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.120654    4519 scope.go:122] "RemoveContainer" containerID="e149d10d744aa0c7cc85b2684ff11cdcd5c4f72d714e69536bea2b5e52db4961"
Oct 31 16:15:08.121954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.121935    4519 generic.go:356] "Generic (PLEG): container finished" podID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerID="740832e9b364b30379e2e87dadf2a9e445c3e0feb4dc263d4c98501cbf3d4e1a" exitCode=60
Oct 31 16:15:08.122039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.122022    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerDied","Data":"740832e9b364b30379e2e87dadf2a9e445c3e0feb4dc263d4c98501cbf3d4e1a"}
Oct 31 16:15:08.122383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.122367    4519 scope.go:122] "RemoveContainer" containerID="740832e9b364b30379e2e87dadf2a9e445c3e0feb4dc263d4c98501cbf3d4e1a"
Oct 31 16:15:08.135794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:08.135776    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:15:09.129860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:09.129831    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerStarted","Data":"794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff"}
Oct 31 16:15:09.131851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:09.131828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerStarted","Data":"aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf"}
Oct 31 16:15:09.133702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:09.133678    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerStarted","Data":"88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277"}
Oct 31 16:15:09.135520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:09.135497    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerStarted","Data":"c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88"}
Oct 31 16:15:09.137224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:09.137204    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerStarted","Data":"861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87"}
Oct 31 16:15:10.145233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.145202    4519 generic.go:356] "Generic (PLEG): container finished" podID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerID="aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf" exitCode=60
Oct 31 16:15:10.145733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.145291    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerDied","Data":"aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf"}
Oct 31 16:15:10.145733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.145366    4519 scope.go:122] "RemoveContainer" containerID="740832e9b364b30379e2e87dadf2a9e445c3e0feb4dc263d4c98501cbf3d4e1a"
Oct 31 16:15:10.145733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.145536    4519 scope.go:122] "RemoveContainer" containerID="aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf"
Oct 31 16:15:10.145900 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:10.145829    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br_odh-ai-gateway-infra(235ee35f-9f11-48fc-a583-1fb22e1e4327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327"
Oct 31 16:15:10.147215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.147195    4519 generic.go:356] "Generic (PLEG): container finished" podID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerID="88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277" exitCode=60
Oct 31 16:15:10.147318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.147262    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerDied","Data":"88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277"}
Oct 31 16:15:10.147499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.147431    4519 scope.go:122] "RemoveContainer" containerID="88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277"
Oct 31 16:15:10.147721 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:10.147692    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps_odh-ai-gateway-infra(b41d42c9-376a-4d61-88b3-a01bb16b31f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4"
Oct 31 16:15:10.149086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.149067    4519 generic.go:356] "Generic (PLEG): container finished" podID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerID="c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88" exitCode=60
Oct 31 16:15:10.149181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.149127    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerDied","Data":"c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88"}
Oct 31 16:15:10.149383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.149364    4519 scope.go:122] "RemoveContainer" containerID="c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88"
Oct 31 16:15:10.149600 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:10.149581    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb_odh-ai-gateway-infra(21e1f774-b10a-4ad8-9f70-cf79ce1669ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab"
Oct 31 16:15:10.151063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.151039    4519 generic.go:356] "Generic (PLEG): container finished" podID="d2570659-f361-408c-b860-fba8eee4763b" containerID="861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87" exitCode=60
Oct 31 16:15:10.151169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.151114    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerDied","Data":"861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87"}
Oct 31 16:15:10.151283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.151270    4519 scope.go:122] "RemoveContainer" containerID="861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87"
Oct 31 16:15:10.151441 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:10.151421    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8_odh-ai-gateway-infra(d2570659-f361-408c-b860-fba8eee4763b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" podUID="d2570659-f361-408c-b860-fba8eee4763b"
Oct 31 16:15:10.152993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.152968    4519 generic.go:356] "Generic (PLEG): container finished" podID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerID="794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff" exitCode=60
Oct 31 16:15:10.153088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.153008    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerDied","Data":"794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff"}
Oct 31 16:15:10.153239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.153224    4519 scope.go:122] "RemoveContainer" containerID="794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff"
Oct 31 16:15:10.153531 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:10.153417    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891055-q8h2v_odh-ai-gateway-infra(42942174-e286-4d8b-bf4b-dc8f07371e13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13"
Oct 31 16:15:10.158055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.158037    4519 scope.go:122] "RemoveContainer" containerID="83f84464f4d02fb774c322c4d4a2e99c237312207fd82eac9ce0af0d289e69bc"
Oct 31 16:15:10.170132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.170110    4519 scope.go:122] "RemoveContainer" containerID="b48e01f506c3b98cde6ff27b4c09bc9c99ed41ebc9788378b94509c51ea28b03"
Oct 31 16:15:10.181775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.181751    4519 scope.go:122] "RemoveContainer" containerID="02e54e87e76080f1dde0b2996dd7b9bea491b7a062177a73ed082b998c9415bc"
Oct 31 16:15:10.193175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:10.193157    4519 scope.go:122] "RemoveContainer" containerID="e149d10d744aa0c7cc85b2684ff11cdcd5c4f72d714e69536bea2b5e52db4961"
Oct 31 16:15:11.161043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:11.161013    4519 scope.go:122] "RemoveContainer" containerID="794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff"
Oct 31 16:15:11.161491 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:11.161275    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891055-q8h2v_odh-ai-gateway-infra(42942174-e286-4d8b-bf4b-dc8f07371e13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13"
Oct 31 16:15:11.162648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:11.162627    4519 scope.go:122] "RemoveContainer" containerID="aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf"
Oct 31 16:15:11.162859 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:11.162840    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br_odh-ai-gateway-infra(235ee35f-9f11-48fc-a583-1fb22e1e4327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327"
Oct 31 16:15:11.164150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:11.164133    4519 scope.go:122] "RemoveContainer" containerID="88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277"
Oct 31 16:15:11.164361 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:11.164340    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps_odh-ai-gateway-infra(b41d42c9-376a-4d61-88b3-a01bb16b31f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4"
Oct 31 16:15:11.165641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:11.165624    4519 scope.go:122] "RemoveContainer" containerID="c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88"
Oct 31 16:15:11.165830 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:11.165811    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb_odh-ai-gateway-infra(21e1f774-b10a-4ad8-9f70-cf79ce1669ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab"
Oct 31 16:15:11.167203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:11.167187    4519 scope.go:122] "RemoveContainer" containerID="861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87"
Oct 31 16:15:11.167346 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:11.167330    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8_odh-ai-gateway-infra(d2570659-f361-408c-b860-fba8eee4763b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" podUID="d2570659-f361-408c-b860-fba8eee4763b"
Oct 31 16:15:20.431749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:20.431660    4519 scope.go:122] "RemoveContainer" containerID="794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff"
Oct 31 16:15:21.228674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:21.228641    4519 generic.go:356] "Generic (PLEG): container finished" podID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerID="4a7353ba2a92724d69524b58d39a8755d65a5f4bc4bbbc8d4c23e9c3db724176" exitCode=60
Oct 31 16:15:21.228841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:21.228712    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerDied","Data":"4a7353ba2a92724d69524b58d39a8755d65a5f4bc4bbbc8d4c23e9c3db724176"}
Oct 31 16:15:21.228841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:21.228771    4519 scope.go:122] "RemoveContainer" containerID="794f8dd9151b7cf8322dd7f020d507d76a602fa3be1fde9eb7ad72dcc7f0adff"
Oct 31 16:15:21.229139 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:21.229120    4519 scope.go:122] "RemoveContainer" containerID="4a7353ba2a92724d69524b58d39a8755d65a5f4bc4bbbc8d4c23e9c3db724176"
Oct 31 16:15:21.229404 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:21.229384    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891055-q8h2v_odh-ai-gateway-infra(42942174-e286-4d8b-bf4b-dc8f07371e13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13"
Oct 31 16:15:21.431734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:21.431659    4519 scope.go:122] "RemoveContainer" containerID="88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277"
Oct 31 16:15:22.237870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.237838    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerStarted","Data":"5de3c93589894b06dc442f1dfef3f09f372b87205ffc258eef7c414c0a8d9d76"}
Oct 31 16:15:22.346631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.346597    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"]
Oct 31 16:15:22.431773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.431744    4519 scope.go:122] "RemoveContainer" containerID="861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87"
Oct 31 16:15:22.440998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.434647    4519 scope.go:122] "RemoveContainer" containerID="c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88"
Oct 31 16:15:22.524035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.524014    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:22.570562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.570535    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca\") pod \"42942174-e286-4d8b-bf4b-dc8f07371e13\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") "
Oct 31 16:15:22.570723 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.570708    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt\" (UniqueName: \"kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt\") pod \"42942174-e286-4d8b-bf4b-dc8f07371e13\" (UID: \"42942174-e286-4d8b-bf4b-dc8f07371e13\") "
Oct 31 16:15:22.570862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.570841    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca" pod "42942174-e286-4d8b-bf4b-dc8f07371e13" (UID: "42942174-e286-4d8b-bf4b-dc8f07371e13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:22.571289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.571262    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42942174-e286-4d8b-bf4b-dc8f07371e13-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:22.572966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.572942    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt" pod "42942174-e286-4d8b-bf4b-dc8f07371e13" (UID: "42942174-e286-4d8b-bf4b-dc8f07371e13"). InnerVolumeSpecName "kube-api-access-mhhbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:22.672732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:22.672698    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhbt\" (UniqueName: \"kubernetes.io/projected/42942174-e286-4d8b-bf4b-dc8f07371e13-kube-api-access-mhhbt\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:23.245319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.245287    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"
Oct 31 16:15:23.245814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.245320    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v" event={"ID":"42942174-e286-4d8b-bf4b-dc8f07371e13","Type":"ContainerDied","Data":"7c0fd6b551d162906b9dc611127cb1b4d3ef141940fa632ae5d2e44d6efabab5"}
Oct 31 16:15:23.245814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.245393    4519 scope.go:122] "RemoveContainer" containerID="4a7353ba2a92724d69524b58d39a8755d65a5f4bc4bbbc8d4c23e9c3db724176"
Oct 31 16:15:23.247383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.247356    4519 generic.go:356] "Generic (PLEG): container finished" podID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerID="5de3c93589894b06dc442f1dfef3f09f372b87205ffc258eef7c414c0a8d9d76" exitCode=60
Oct 31 16:15:23.247515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.247419    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerDied","Data":"5de3c93589894b06dc442f1dfef3f09f372b87205ffc258eef7c414c0a8d9d76"}
Oct 31 16:15:23.247793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.247777    4519 scope.go:122] "RemoveContainer" containerID="5de3c93589894b06dc442f1dfef3f09f372b87205ffc258eef7c414c0a8d9d76"
Oct 31 16:15:23.248071 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:23.248048    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps_odh-ai-gateway-infra(b41d42c9-376a-4d61-88b3-a01bb16b31f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4"
Oct 31 16:15:23.249637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.249613    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerStarted","Data":"3e9bc72a365568b2bf2a1748b596057b8d80997661269ea43af73dd1cb0993be"}
Oct 31 16:15:23.251550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.251530    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerStarted","Data":"78cc824c470037c4eb35a5caee37f76cae6e5903d3c8284a712896c3b1f36f47"}
Oct 31 16:15:23.255745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.255730    4519 scope.go:122] "RemoveContainer" containerID="88534549661cefbb5c6ff067232a89219be8f7f0dc417b6f8a42cd2e8b1ef277"
Oct 31 16:15:23.356640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.356604    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"]
Oct 31 16:15:23.431887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.431868    4519 scope.go:122] "RemoveContainer" containerID="aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf"
Oct 31 16:15:23.615635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.615600    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"]
Oct 31 16:15:23.620857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:23.620830    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-q8h2v"]
Oct 31 16:15:24.004113 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:24.004077    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:15:24.259324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.259295    4519 generic.go:356] "Generic (PLEG): container finished" podID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerID="6ce631cfa9402668cc8caddea70700fdd016c4613734dad378603335290028c4" exitCode=60
Oct 31 16:15:24.259787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.259366    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerDied","Data":"6ce631cfa9402668cc8caddea70700fdd016c4613734dad378603335290028c4"}
Oct 31 16:15:24.259787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.259420    4519 scope.go:122] "RemoveContainer" containerID="aff632e1792920d574d1b5e0fd311200e7ea9e5948049129632ee76c3564f7bf"
Oct 31 16:15:24.259887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.259836    4519 scope.go:122] "RemoveContainer" containerID="6ce631cfa9402668cc8caddea70700fdd016c4613734dad378603335290028c4"
Oct 31 16:15:24.260191 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:24.260162    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br_odh-ai-gateway-infra(235ee35f-9f11-48fc-a583-1fb22e1e4327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327"
Oct 31 16:15:24.262833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.262807    4519 generic.go:356] "Generic (PLEG): container finished" podID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerID="3e9bc72a365568b2bf2a1748b596057b8d80997661269ea43af73dd1cb0993be" exitCode=60
Oct 31 16:15:24.262918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.262870    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerDied","Data":"3e9bc72a365568b2bf2a1748b596057b8d80997661269ea43af73dd1cb0993be"}
Oct 31 16:15:24.263194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.263178    4519 scope.go:122] "RemoveContainer" containerID="3e9bc72a365568b2bf2a1748b596057b8d80997661269ea43af73dd1cb0993be"
Oct 31 16:15:24.263485 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:24.263438    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb_odh-ai-gateway-infra(21e1f774-b10a-4ad8-9f70-cf79ce1669ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab"
Oct 31 16:15:24.264936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.264915    4519 generic.go:356] "Generic (PLEG): container finished" podID="d2570659-f361-408c-b860-fba8eee4763b" containerID="78cc824c470037c4eb35a5caee37f76cae6e5903d3c8284a712896c3b1f36f47" exitCode=60
Oct 31 16:15:24.265010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.264988    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerDied","Data":"78cc824c470037c4eb35a5caee37f76cae6e5903d3c8284a712896c3b1f36f47"}
Oct 31 16:15:24.265331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.265316    4519 scope.go:122] "RemoveContainer" containerID="78cc824c470037c4eb35a5caee37f76cae6e5903d3c8284a712896c3b1f36f47"
Oct 31 16:15:24.265565 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:24.265543    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8_odh-ai-gateway-infra(d2570659-f361-408c-b860-fba8eee4763b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" podUID="d2570659-f361-408c-b860-fba8eee4763b"
Oct 31 16:15:24.350376 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.350345    4519 scope.go:122] "RemoveContainer" containerID="c52c5adf10fac536992d677792277617d5850c27000cf3949b14b1fd76765d88"
Oct 31 16:15:24.363518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.363494    4519 scope.go:122] "RemoveContainer" containerID="861bba85d14bb766b1e2bd8e620d468c3d2e711eea7dbacbeadb1443b157cc87"
Oct 31 16:15:24.409931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.409904    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:24.410135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.410116    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"]
Oct 31 16:15:24.438855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.438784    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" path="/var/lib/kubelet/pods/42942174-e286-4d8b-bf4b-dc8f07371e13/volumes"
Oct 31 16:15:24.493860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.493830    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca\") pod \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") "
Oct 31 16:15:24.494019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.493874    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4\" (UniqueName: \"kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4\") pod \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\" (UID: \"b41d42c9-376a-4d61-88b3-a01bb16b31f4\") "
Oct 31 16:15:24.494193 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.494169    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca" pod "b41d42c9-376a-4d61-88b3-a01bb16b31f4" (UID: "b41d42c9-376a-4d61-88b3-a01bb16b31f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:24.494305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.494290    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41d42c9-376a-4d61-88b3-a01bb16b31f4-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:24.495949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.495928    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4" pod "b41d42c9-376a-4d61-88b3-a01bb16b31f4" (UID: "b41d42c9-376a-4d61-88b3-a01bb16b31f4"). InnerVolumeSpecName "kube-api-access-fhkj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:24.595382 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.595352    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhkj4\" (UniqueName: \"kubernetes.io/projected/b41d42c9-376a-4d61-88b3-a01bb16b31f4-kube-api-access-fhkj4\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:24.734923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:24.734891    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"]
Oct 31 16:15:25.275710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.275673    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps" event={"ID":"b41d42c9-376a-4d61-88b3-a01bb16b31f4","Type":"ContainerDied","Data":"0cec89922cb552b18b1c5bfb5bc5fedcaef89756314dcc42419d2ed4264c8dae"}
Oct 31 16:15:25.275710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.275696    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"
Oct 31 16:15:25.276370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.275738    4519 scope.go:122] "RemoveContainer" containerID="5de3c93589894b06dc442f1dfef3f09f372b87205ffc258eef7c414c0a8d9d76"
Oct 31 16:15:25.377148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.377117    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"]
Oct 31 16:15:25.387840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.387807    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:15:25.445665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.443433    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:25.479432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.479405    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:25.504060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.504027    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca\") pod \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") "
Oct 31 16:15:25.504228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.504091    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6\" (UniqueName: \"kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6\") pod \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\" (UID: \"21e1f774-b10a-4ad8-9f70-cf79ce1669ab\") "
Oct 31 16:15:25.504469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.504421    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca" pod "21e1f774-b10a-4ad8-9f70-cf79ce1669ab" (UID: "21e1f774-b10a-4ad8-9f70-cf79ce1669ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:25.504786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.504764    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.506505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.506480    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6" pod "21e1f774-b10a-4ad8-9f70-cf79ce1669ab" (UID: "21e1f774-b10a-4ad8-9f70-cf79ce1669ab"). InnerVolumeSpecName "kube-api-access-2znd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:25.541059 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.541034    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:25.542356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.542332    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"]
Oct 31 16:15:25.545357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.545335    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891055-267ps"]
Oct 31 16:15:25.605817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.605785    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca\") pod \"235ee35f-9f11-48fc-a583-1fb22e1e4327\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") "
Oct 31 16:15:25.605972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.605837    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r\" (UniqueName: \"kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r\") pod \"d2570659-f361-408c-b860-fba8eee4763b\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") "
Oct 31 16:15:25.605972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.605874    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t\" (UniqueName: \"kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t\") pod \"235ee35f-9f11-48fc-a583-1fb22e1e4327\" (UID: \"235ee35f-9f11-48fc-a583-1fb22e1e4327\") "
Oct 31 16:15:25.605972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.605949    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca\") pod \"d2570659-f361-408c-b860-fba8eee4763b\" (UID: \"d2570659-f361-408c-b860-fba8eee4763b\") "
Oct 31 16:15:25.606268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.606247    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2znd6\" (UniqueName: \"kubernetes.io/projected/21e1f774-b10a-4ad8-9f70-cf79ce1669ab-kube-api-access-2znd6\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.606383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.606286    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca" pod "235ee35f-9f11-48fc-a583-1fb22e1e4327" (UID: "235ee35f-9f11-48fc-a583-1fb22e1e4327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:25.606425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.606398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca" pod "d2570659-f361-408c-b860-fba8eee4763b" (UID: "d2570659-f361-408c-b860-fba8eee4763b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:25.608057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.608036    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t" pod "235ee35f-9f11-48fc-a583-1fb22e1e4327" (UID: "235ee35f-9f11-48fc-a583-1fb22e1e4327"). InnerVolumeSpecName "kube-api-access-cks7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:25.608125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.608074    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r" pod "d2570659-f361-408c-b860-fba8eee4763b" (UID: "d2570659-f361-408c-b860-fba8eee4763b"). InnerVolumeSpecName "kube-api-access-2gf8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:25.706932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.706898    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ee35f-9f11-48fc-a583-1fb22e1e4327-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.706932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.706926    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gf8r\" (UniqueName: \"kubernetes.io/projected/d2570659-f361-408c-b860-fba8eee4763b-kube-api-access-2gf8r\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.706932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.706936    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cks7t\" (UniqueName: \"kubernetes.io/projected/235ee35f-9f11-48fc-a583-1fb22e1e4327-kube-api-access-cks7t\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.707191 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:25.706944    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2570659-f361-408c-b860-fba8eee4763b-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:26.286135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.286106    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"
Oct 31 16:15:26.286135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.286114    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8" event={"ID":"d2570659-f361-408c-b860-fba8eee4763b","Type":"ContainerDied","Data":"6260c162bd0fdd9a9511945f47e104bdf6707db8bbaa16b73cc2269d2132c1c6"}
Oct 31 16:15:26.286649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.286175    4519 scope.go:122] "RemoveContainer" containerID="78cc824c470037c4eb35a5caee37f76cae6e5903d3c8284a712896c3b1f36f47"
Oct 31 16:15:26.287697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.287676    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"
Oct 31 16:15:26.287782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.287701    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br" event={"ID":"235ee35f-9f11-48fc-a583-1fb22e1e4327","Type":"ContainerDied","Data":"4fe8213b1c0a14bbf7d97321f6860aa9a2b30877f407b4d948cfdf9c1779edcb"}
Oct 31 16:15:26.290180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.290155    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb" event={"ID":"21e1f774-b10a-4ad8-9f70-cf79ce1669ab","Type":"ContainerDied","Data":"308fd237b2c7fb4bb115a2815e452551c3c4d4b3b1eaa5bd2984c87c694f8e6f"}
Oct 31 16:15:26.290307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.290165    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"
Oct 31 16:15:26.298020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.297998    4519 scope.go:122] "RemoveContainer" containerID="6ce631cfa9402668cc8caddea70700fdd016c4613734dad378603335290028c4"
Oct 31 16:15:26.309764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.309744    4519 scope.go:122] "RemoveContainer" containerID="3e9bc72a365568b2bf2a1748b596057b8d80997661269ea43af73dd1cb0993be"
Oct 31 16:15:26.439380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.439346    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" path="/var/lib/kubelet/pods/b41d42c9-376a-4d61-88b3-a01bb16b31f4/volumes"
Oct 31 16:15:26.442734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.442712    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"]
Oct 31 16:15:26.448802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.448781    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891055-6z9br"]
Oct 31 16:15:26.552176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.552095    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"]
Oct 31 16:15:26.554418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.554394    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891055-cxdx8"]
Oct 31 16:15:26.656793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.656760    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"]
Oct 31 16:15:26.660626 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:26.660597    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891055-q95cb"]
Oct 31 16:15:28.438118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:28.438074    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" path="/var/lib/kubelet/pods/21e1f774-b10a-4ad8-9f70-cf79ce1669ab/volumes"
Oct 31 16:15:28.438542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:28.438472    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" path="/var/lib/kubelet/pods/235ee35f-9f11-48fc-a583-1fb22e1e4327/volumes"
Oct 31 16:15:28.438832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:28.438818    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2570659-f361-408c-b860-fba8eee4763b" path="/var/lib/kubelet/pods/d2570659-f361-408c-b860-fba8eee4763b/volumes"
Oct 31 16:15:35.786058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786021    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"]
Oct 31 16:15:35.786569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786553    4519 cpu_manager.go:414] "containerMap: removing container" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:35.786618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786572    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:35.786618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786588    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.786618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786604    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.786618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786617    4519 cpu_manager.go:414] "containerMap: removing container" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786622    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786636    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786640    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786650    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786655    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786666    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786670    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786684    4519 cpu_manager.go:414] "containerMap: removing container" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786689    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786695    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786699    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786712    4519 cpu_manager.go:414] "containerMap: removing container" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786716    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786724    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786728    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786736    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786740    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786750    4519 cpu_manager.go:414] "containerMap: removing container" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.786756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786754    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786832    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786840    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786845    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786852    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786858    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786864    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786870    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786876    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786881    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786886    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786891    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="235ee35f-9f11-48fc-a583-1fb22e1e4327" containerName="cleanup"
Oct 31 16:15:35.787291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.786897    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:35.791883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.791867    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:35.841818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.841786    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"openshift-service-ca.crt\""
Oct 31 16:15:35.841818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.841786    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"kube-root-ca.crt\""
Oct 31 16:15:35.888998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.888969    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"premium-w5-ef6d93-kserve-self-signed-certs\""
Oct 31 16:15:35.889179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.889137    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"default-dockercfg-hnnx7\""
Oct 31 16:15:35.891481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.891448    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"]
Oct 31 16:15:35.917087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.917063    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:35.917169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.917116    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:35.917281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.917260    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:35.917366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.917303    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:35.917400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.917365    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5z2\" (UniqueName: \"kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:35.917433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:35.917405    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019038    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019086    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019109    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5z2\" (UniqueName: \"kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019127    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019174    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019230    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019612    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019647    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.019766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.019693    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.021348 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.021326    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.021655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.021634    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.073161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.073093    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5z2\" (UniqueName: \"kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2\") pod \"premium-w5-ef6d93-kserve-7c9c4645d-74nm6\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") " pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.103246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.103220    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:36.489741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:36.489707    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"]
Oct 31 16:15:36.496033 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:36.496003    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7073ebb0_d92a_4286_bd81_23a6800f9425.slice/crio-3c25067d38ff13510e39f7b4210e711259c2fdca5db901f9169d157a7b4daade WatchSource:0}: Error finding container 3c25067d38ff13510e39f7b4210e711259c2fdca5db901f9169d157a7b4daade: Status 404 returned error can't find the container with id 3c25067d38ff13510e39f7b4210e711259c2fdca5db901f9169d157a7b4daade
Oct 31 16:15:37.355260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:37.355226    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" event={"ID":"7073ebb0-d92a-4286-bd81-23a6800f9425","Type":"ContainerStarted","Data":"1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69"}
Oct 31 16:15:37.355656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:37.355267    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" event={"ID":"7073ebb0-d92a-4286-bd81-23a6800f9425","Type":"ContainerStarted","Data":"3c25067d38ff13510e39f7b4210e711259c2fdca5db901f9169d157a7b4daade"}
Oct 31 16:15:37.478708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:37.478649    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podStartSLOduration=2.478635244 podStartE2EDuration="2.478635244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:37.477168194 +0000 UTC m=+781.177097505" watchObservedRunningTime="2026-10-31 16:15:37.478635244 +0000 UTC m=+781.178564554"
Oct 31 16:15:38.103643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:38.103606    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:45.964811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:45.964776    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:15:46.103708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:46.103675    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:46.116410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:46.116386    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:46.416799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:46.416724    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:15:52.279946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.279914    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"]
Oct 31 16:15:52.280443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280431    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:52.280532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280445    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:52.280532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280497    4519 cpu_manager.go:414] "containerMap: removing container" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:52.280532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280506    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:52.280532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280532    4519 cpu_manager.go:414] "containerMap: removing container" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:52.280669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280538    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42942174-e286-4d8b-bf4b-dc8f07371e13" containerName="cleanup"
Oct 31 16:15:52.280669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280625    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41d42c9-376a-4d61-88b3-a01bb16b31f4" containerName="cleanup"
Oct 31 16:15:52.280669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280638    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e1f774-b10a-4ad8-9f70-cf79ce1669ab" containerName="cleanup"
Oct 31 16:15:52.280669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.280644    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2570659-f361-408c-b860-fba8eee4763b" containerName="cleanup"
Oct 31 16:15:52.283892 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.283876    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.336656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.336622    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"]
Oct 31 16:15:52.380434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.380407    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"premium-w0-ac1273-kserve-self-signed-certs\""
Oct 31 16:15:52.394052 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.394025    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.394175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.394159    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.394234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.394185    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.394329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.394308    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.394417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.394401    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.394533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.394518    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7gc\" (UniqueName: \"kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.495339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495299    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7gc\" (UniqueName: \"kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.495533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495362    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.495533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495487    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.495533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495515    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.495674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495565    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.495674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495623    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.496030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.495996    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.496164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.496036    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.496164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.496070    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.498284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.498256    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.499547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.498555    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.548905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.548844    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7gc\" (UniqueName: \"kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc\") pod \"premium-w0-ac1273-kserve-55d89dc5bf-qzg7b\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") " pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.596877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.596845    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:52.981589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:52.981550    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"]
Oct 31 16:15:52.986912 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:15:52.986883    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbec0b6b0_05b2_42a1_9759_b765b6126855.slice/crio-080299d172a1b99c3751c59b7bc0bc8ef60c6e7266268d2ace655618404b1fdc WatchSource:0}: Error finding container 080299d172a1b99c3751c59b7bc0bc8ef60c6e7266268d2ace655618404b1fdc: Status 404 returned error can't find the container with id 080299d172a1b99c3751c59b7bc0bc8ef60c6e7266268d2ace655618404b1fdc
Oct 31 16:15:53.453331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:53.453240    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" event={"ID":"bec0b6b0-05b2-42a1-9759-b765b6126855","Type":"ContainerStarted","Data":"468c02d75ce77541ca7b250f5555f69023bd98d5f1a93052b884e0925fe6f818"}
Oct 31 16:15:53.453331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:53.453279    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" event={"ID":"bec0b6b0-05b2-42a1-9759-b765b6126855","Type":"ContainerStarted","Data":"080299d172a1b99c3751c59b7bc0bc8ef60c6e7266268d2ace655618404b1fdc"}
Oct 31 16:15:53.562675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:53.562626    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podStartSLOduration=1.562612159 podStartE2EDuration="1.562612159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:53.560799134 +0000 UTC m=+797.260728446" watchObservedRunningTime="2026-10-31 16:15:53.562612159 +0000 UTC m=+797.262541469"
Oct 31 16:15:53.597230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:15:53.597199    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:15:54.005084 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:15:54.005051    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:16:02.597044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:02.597011    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:16:02.609797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:02.609769    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:16:03.516283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:03.516247    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:16:14.062024 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:14.061988    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:16:20.278487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.278442    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"]
Oct 31 16:16:20.283253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.283234    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.333826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.333782    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"]
Oct 31 16:16:20.378285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.378246    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"openshift-service-ca.crt\""
Oct 31 16:16:20.378285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.378286    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"kube-root-ca.crt\""
Oct 31 16:16:20.378510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.378391    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"premium-w2-e7f13a-kserve-self-signed-certs\""
Oct 31 16:16:20.380198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.380183    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"default-dockercfg-mj79l\""
Oct 31 16:16:20.412798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.412765    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.412798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.412800    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.413000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.412820    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkhj\" (UniqueName: \"kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.413000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.412965    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.413000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.412990    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.413129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.413081    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514195    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514229    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514250    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkhj\" (UniqueName: \"kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514304    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514327    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514403    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514747    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.514890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.514793    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.516659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.516630    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.516898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.516882    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.568583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.568514    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkhj\" (UniqueName: \"kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj\") pod \"premium-w2-e7f13a-kserve-65fd447687-7gn44\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") " pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.596611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.596584    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:20.974931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:20.974899    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"]
Oct 31 16:16:20.980787 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:16:20.980758    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b495c62_d368_4270_9576_f59625676332.slice/crio-0d385575160d2812b7cf24ffdaa8e4907648223a9598f207bf4ce010c57a8c5c WatchSource:0}: Error finding container 0d385575160d2812b7cf24ffdaa8e4907648223a9598f207bf4ce010c57a8c5c: Status 404 returned error can't find the container with id 0d385575160d2812b7cf24ffdaa8e4907648223a9598f207bf4ce010c57a8c5c
Oct 31 16:16:21.625664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:21.625628    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" event={"ID":"3b495c62-d368-4270-9576-f59625676332","Type":"ContainerStarted","Data":"69fe5be07cfe39afbf3bd1d3dc88f4d8b0f43f83e5cd8023c8b5a5cab7fa9ab4"}
Oct 31 16:16:21.626065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:21.625676    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" event={"ID":"3b495c62-d368-4270-9576-f59625676332","Type":"ContainerStarted","Data":"0d385575160d2812b7cf24ffdaa8e4907648223a9598f207bf4ce010c57a8c5c"}
Oct 31 16:16:21.735226 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:21.735176    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podStartSLOduration=1.735164482 podStartE2EDuration="1.735164482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:21.733759655 +0000 UTC m=+825.433688966" watchObservedRunningTime="2026-10-31 16:16:21.735164482 +0000 UTC m=+825.435093792"
Oct 31 16:16:22.597783 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:22.597749    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:24.004183 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:16:24.004137    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:16:30.597343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:30.597310    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:30.610216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:30.610190    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:30.687696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:30.687670    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:16:32.301237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.301204    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"]
Oct 31 16:16:32.305412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.305395    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.345768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.345739    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.345907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.345780    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvmq\" (UniqueName: \"kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.345907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.345805    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.355942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.355888    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:16:32.357956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.357932    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"]
Oct 31 16:16:32.379034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.379011    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:16:32.379034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.379013    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:16:32.401898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.401875    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:16:32.447372 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.447344    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.447540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.447500    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.447540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.447526    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvmq\" (UniqueName: \"kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.448043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.448019    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.449776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.449757    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.501157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.501126    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvmq\" (UniqueName: \"kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq\") pod \"maas-api-revoke-sub-2912257c-txmb2\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:32.618369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:32.618297    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:33.000868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:33.000829    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"]
Oct 31 16:16:33.006536 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:16:33.006507    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3502a88c_3568_4a72_8e5a_36bdd99914d6.slice/crio-33fc32f1e487e9010991fa244ad22e5128beedbe8b36d9b2a65abfad7d259df7 WatchSource:0}: Error finding container 33fc32f1e487e9010991fa244ad22e5128beedbe8b36d9b2a65abfad7d259df7: Status 404 returned error can't find the container with id 33fc32f1e487e9010991fa244ad22e5128beedbe8b36d9b2a65abfad7d259df7
Oct 31 16:16:33.700183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:33.700092    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2" event={"ID":"3502a88c-3568-4a72-8e5a-36bdd99914d6","Type":"ContainerStarted","Data":"e413e46a6faa2b85bdabfa6a49cc53f74d2605c76c1f056df90b7710c551efc7"}
Oct 31 16:16:33.700183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:33.700145    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2" event={"ID":"3502a88c-3568-4a72-8e5a-36bdd99914d6","Type":"ContainerStarted","Data":"33fc32f1e487e9010991fa244ad22e5128beedbe8b36d9b2a65abfad7d259df7"}
Oct 31 16:16:33.808626 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:33.808575    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2" podStartSLOduration=1.808562401 podStartE2EDuration="1.808562401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:33.806680946 +0000 UTC m=+837.506610257" watchObservedRunningTime="2026-10-31 16:16:33.808562401 +0000 UTC m=+837.508491711"
Oct 31 16:16:34.707510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:34.707449    4519 generic.go:356] "Generic (PLEG): container finished" podID="3502a88c-3568-4a72-8e5a-36bdd99914d6" containerID="e413e46a6faa2b85bdabfa6a49cc53f74d2605c76c1f056df90b7710c551efc7" exitCode=0
Oct 31 16:16:34.707905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:34.707523    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2" event={"ID":"3502a88c-3568-4a72-8e5a-36bdd99914d6","Type":"ContainerDied","Data":"e413e46a6faa2b85bdabfa6a49cc53f74d2605c76c1f056df90b7710c551efc7"}
Oct 31 16:16:35.860761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.860735    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:35.990882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.990811    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq\" (UniqueName: \"kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq\") pod \"3502a88c-3568-4a72-8e5a-36bdd99914d6\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") "
Oct 31 16:16:35.991030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.990942    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca\") pod \"3502a88c-3568-4a72-8e5a-36bdd99914d6\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") "
Oct 31 16:16:35.991030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.990976    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca\") pod \"3502a88c-3568-4a72-8e5a-36bdd99914d6\" (UID: \"3502a88c-3568-4a72-8e5a-36bdd99914d6\") "
Oct 31 16:16:35.991336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.991311    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca" pod "3502a88c-3568-4a72-8e5a-36bdd99914d6" (UID: "3502a88c-3568-4a72-8e5a-36bdd99914d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:16:35.991491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.991450    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:35.993419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.993389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca" pod "3502a88c-3568-4a72-8e5a-36bdd99914d6" (UID: "3502a88c-3568-4a72-8e5a-36bdd99914d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:16:35.993504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:35.993412    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq" pod "3502a88c-3568-4a72-8e5a-36bdd99914d6" (UID: "3502a88c-3568-4a72-8e5a-36bdd99914d6"). InnerVolumeSpecName "kube-api-access-nlvmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:16:36.092438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:36.092397    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvmq\" (UniqueName: \"kubernetes.io/projected/3502a88c-3568-4a72-8e5a-36bdd99914d6-kube-api-access-nlvmq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:36.092438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:36.092436    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3502a88c-3568-4a72-8e5a-36bdd99914d6-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:36.721046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:36.721021    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"
Oct 31 16:16:36.721046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:36.721025    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2" event={"ID":"3502a88c-3568-4a72-8e5a-36bdd99914d6","Type":"ContainerDied","Data":"33fc32f1e487e9010991fa244ad22e5128beedbe8b36d9b2a65abfad7d259df7"}
Oct 31 16:16:36.721258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:36.721078    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33fc32f1e487e9010991fa244ad22e5128beedbe8b36d9b2a65abfad7d259df7"
Oct 31 16:16:47.729213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.729134    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"]
Oct 31 16:16:47.729790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.729769    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3502a88c-3568-4a72-8e5a-36bdd99914d6" containerName="revoke-keys"
Oct 31 16:16:47.729790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.729791    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3502a88c-3568-4a72-8e5a-36bdd99914d6" containerName="revoke-keys"
Oct 31 16:16:47.729910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.729900    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3502a88c-3568-4a72-8e5a-36bdd99914d6" containerName="revoke-keys"
Oct 31 16:16:47.733427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.733405    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:47.787109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.787074    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"]
Oct 31 16:16:47.830656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.830625    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:16:47.830656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.830646    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:16:47.830656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.830625    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:16:47.830829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.830679    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:16:47.923860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.923822    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:47.924026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.923872    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7sb\" (UniqueName: \"kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:47.924071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:47.924049    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.025177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.025093    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.025177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.025139    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7sb\" (UniqueName: \"kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.025361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.025269    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.025918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.025899    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.027602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.027584    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.078955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.078927    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7sb\" (UniqueName: \"kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb\") pod \"maas-api-revoke-sub-41a120d8-r6rqt\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.364278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.364205    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:48.754637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.754605    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:16:48.761606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.761583    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"]
Oct 31 16:16:48.767159 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:16:48.767132    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9e1cf1_a092_4c34_b0d8_5994e9db4f8a.slice/crio-9737baa05cf1d07a1f1e7af58d0eb4d31ec469bc2355a91c02480ff966fb8499 WatchSource:0}: Error finding container 9737baa05cf1d07a1f1e7af58d0eb4d31ec469bc2355a91c02480ff966fb8499: Status 404 returned error can't find the container with id 9737baa05cf1d07a1f1e7af58d0eb4d31ec469bc2355a91c02480ff966fb8499
Oct 31 16:16:48.791316 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:48.791287    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt" event={"ID":"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a","Type":"ContainerStarted","Data":"9737baa05cf1d07a1f1e7af58d0eb4d31ec469bc2355a91c02480ff966fb8499"}
Oct 31 16:16:49.800305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:49.800262    4519 generic.go:356] "Generic (PLEG): container finished" podID="cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" containerID="8f32090c6cc0149db9c814603e19b5b2b0995412bb5a6dd3d80b467316fe6a87" exitCode=0
Oct 31 16:16:49.800746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:49.800342    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt" event={"ID":"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a","Type":"ContainerDied","Data":"8f32090c6cc0149db9c814603e19b5b2b0995412bb5a6dd3d80b467316fe6a87"}
Oct 31 16:16:50.964201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:50.964174    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:51.062769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.062686    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca\") pod \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") "
Oct 31 16:16:51.062769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.062727    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb\" (UniqueName: \"kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb\") pod \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") "
Oct 31 16:16:51.062996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.062783    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca\") pod \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\" (UID: \"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a\") "
Oct 31 16:16:51.063065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.063032    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca" pod "cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" (UID: "cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:16:51.063190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.063174    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:51.064921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.064901    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca" pod "cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" (UID: "cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:16:51.064921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.064911    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb" pod "cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" (UID: "cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a"). InnerVolumeSpecName "kube-api-access-cg7sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:16:51.164324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.164285    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7sb\" (UniqueName: \"kubernetes.io/projected/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-kube-api-access-cg7sb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:51.164324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.164318    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:51.813045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.813014    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"
Oct 31 16:16:51.813239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.813050    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt" event={"ID":"cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a","Type":"ContainerDied","Data":"9737baa05cf1d07a1f1e7af58d0eb4d31ec469bc2355a91c02480ff966fb8499"}
Oct 31 16:16:51.813239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:51.813105    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9737baa05cf1d07a1f1e7af58d0eb4d31ec469bc2355a91c02480ff966fb8499"
Oct 31 16:16:54.004487 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:16:54.004436    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:16:54.803520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.803480    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"]
Oct 31 16:16:54.804006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.803994    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" containerName="revoke-keys"
Oct 31 16:16:54.804060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.804008    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" containerName="revoke-keys"
Oct 31 16:16:54.804128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.804119    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" containerName="revoke-keys"
Oct 31 16:16:54.807673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.807653    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:54.861149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.861117    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"]
Oct 31 16:16:54.903877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.903837    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:16:54.903877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.903871    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:16:54.903877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.903876    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:16:54.904782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.904763    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:16:54.910948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.910920    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:54.911060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.911043    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52h2\" (UniqueName: \"kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:54.911118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:54.911089    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.012576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.012542    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52h2\" (UniqueName: \"kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.012954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.012597    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.012954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.012693    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.013397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.013376    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.014907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.014884    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.066517 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.066419    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52h2\" (UniqueName: \"kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2\") pod \"maas-api-revoke-sub-5c0999d3-dvpp5\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.120596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.120563    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:55.489940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.489898    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"]
Oct 31 16:16:55.496153 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:16:55.496125    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ffee20_bf1e_4fbf_9bce_2d74736d8544.slice/crio-7826399ef823b7011ebed41323406923e50c392ae9310f37a5c0cd72aa2b8369 WatchSource:0}: Error finding container 7826399ef823b7011ebed41323406923e50c392ae9310f37a5c0cd72aa2b8369: Status 404 returned error can't find the container with id 7826399ef823b7011ebed41323406923e50c392ae9310f37a5c0cd72aa2b8369
Oct 31 16:16:55.838415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.838378    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5" event={"ID":"e6ffee20-bf1e-4fbf-9bce-2d74736d8544","Type":"ContainerStarted","Data":"d6611833883511d40baeb7daf22ba8b5a799cc40725c72dab7c5fb51e81c77fc"}
Oct 31 16:16:55.838612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.838441    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5" event={"ID":"e6ffee20-bf1e-4fbf-9bce-2d74736d8544","Type":"ContainerStarted","Data":"7826399ef823b7011ebed41323406923e50c392ae9310f37a5c0cd72aa2b8369"}
Oct 31 16:16:55.948971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:55.948906    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5" podStartSLOduration=1.948888947 podStartE2EDuration="1.948888947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:55.944905549 +0000 UTC m=+859.644834861" watchObservedRunningTime="2026-10-31 16:16:55.948888947 +0000 UTC m=+859.648818261"
Oct 31 16:16:56.852766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:56.852732    4519 generic.go:356] "Generic (PLEG): container finished" podID="e6ffee20-bf1e-4fbf-9bce-2d74736d8544" containerID="d6611833883511d40baeb7daf22ba8b5a799cc40725c72dab7c5fb51e81c77fc" exitCode=0
Oct 31 16:16:56.853137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:56.852817    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5" event={"ID":"e6ffee20-bf1e-4fbf-9bce-2d74736d8544","Type":"ContainerDied","Data":"d6611833883511d40baeb7daf22ba8b5a799cc40725c72dab7c5fb51e81c77fc"}
Oct 31 16:16:57.997215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:57.997187    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:58.046390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.046356    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca\") pod \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") "
Oct 31 16:16:58.046590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.046433    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca\") pod \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") "
Oct 31 16:16:58.046590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.046509    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2\" (UniqueName: \"kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2\") pod \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\" (UID: \"e6ffee20-bf1e-4fbf-9bce-2d74736d8544\") "
Oct 31 16:16:58.046794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.046772    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca" pod "e6ffee20-bf1e-4fbf-9bce-2d74736d8544" (UID: "e6ffee20-bf1e-4fbf-9bce-2d74736d8544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:16:58.046932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.046915    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:58.048599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.048572    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca" pod "e6ffee20-bf1e-4fbf-9bce-2d74736d8544" (UID: "e6ffee20-bf1e-4fbf-9bce-2d74736d8544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:16:58.048599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.048575    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2" pod "e6ffee20-bf1e-4fbf-9bce-2d74736d8544" (UID: "e6ffee20-bf1e-4fbf-9bce-2d74736d8544"). InnerVolumeSpecName "kube-api-access-g52h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:16:58.147885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.147802    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:58.147885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.147832    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52h2\" (UniqueName: \"kubernetes.io/projected/e6ffee20-bf1e-4fbf-9bce-2d74736d8544-kube-api-access-g52h2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:16:58.865623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.865593    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"
Oct 31 16:16:58.865623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.865615    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5" event={"ID":"e6ffee20-bf1e-4fbf-9bce-2d74736d8544","Type":"ContainerDied","Data":"7826399ef823b7011ebed41323406923e50c392ae9310f37a5c0cd72aa2b8369"}
Oct 31 16:16:58.865900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:16:58.865663    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7826399ef823b7011ebed41323406923e50c392ae9310f37a5c0cd72aa2b8369"
Oct 31 16:17:03.393488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.393441    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"]
Oct 31 16:17:03.393982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.393969    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e6ffee20-bf1e-4fbf-9bce-2d74736d8544" containerName="revoke-keys"
Oct 31 16:17:03.394079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.393984    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ffee20-bf1e-4fbf-9bce-2d74736d8544" containerName="revoke-keys"
Oct 31 16:17:03.394118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.394092    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ffee20-bf1e-4fbf-9bce-2d74736d8544" containerName="revoke-keys"
Oct 31 16:17:03.397676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.397660    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.448018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.447967    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:17:03.450894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.450863    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"]
Oct 31 16:17:03.495060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.495028    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:17:03.495229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.495087    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:17:03.495568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.495550    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:17:03.505727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.505689    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.505883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.505778    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8wb\" (UniqueName: \"kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.506001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.505977    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.607551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.607513    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.607715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.607571    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.607715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.607606    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8wb\" (UniqueName: \"kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.608170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.608147    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.609891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.609869    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.661829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.661758    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8wb\" (UniqueName: \"kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb\") pod \"maas-api-revoke-sub-76bd0183-bmdlt\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:03.710608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:03.710567    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:04.080777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:04.080739    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"]
Oct 31 16:17:04.086772 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:04.086746    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5189a7_0635_46b2_9b7e_f5601e684119.slice/crio-4cd86c39a29f045f0176bb5463dc7539c375d4f335fda2f86b9358e150facd4e WatchSource:0}: Error finding container 4cd86c39a29f045f0176bb5463dc7539c375d4f335fda2f86b9358e150facd4e: Status 404 returned error can't find the container with id 4cd86c39a29f045f0176bb5463dc7539c375d4f335fda2f86b9358e150facd4e
Oct 31 16:17:04.901499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:04.901450    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt" event={"ID":"8f5189a7-0635-46b2-9b7e-f5601e684119","Type":"ContainerStarted","Data":"df1bf362917cf228b6ab008a13022a2352f9c31d492cb025c4108fac2306806d"}
Oct 31 16:17:04.901987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:04.901525    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt" event={"ID":"8f5189a7-0635-46b2-9b7e-f5601e684119","Type":"ContainerStarted","Data":"4cd86c39a29f045f0176bb5463dc7539c375d4f335fda2f86b9358e150facd4e"}
Oct 31 16:17:05.009339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.009280    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt" podStartSLOduration=2.00926555 podStartE2EDuration="2.00926555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:05.008240781 +0000 UTC m=+868.708170089" watchObservedRunningTime="2026-10-31 16:17:05.00926555 +0000 UTC m=+868.709194861"
Oct 31 16:17:05.604705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.604671    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"]
Oct 31 16:17:05.608681 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.608661    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.664101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.664068    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"]
Oct 31 16:17:05.733765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.733723    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.733923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.733849    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.733923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.733888    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8n9n\" (UniqueName: \"kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.834720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.834683    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.834887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.834763    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.834887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.834786    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8n9n\" (UniqueName: \"kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.835479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.835422    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.837046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.837028    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.888824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.888752    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8n9n\" (UniqueName: \"kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n\") pod \"maas-api-revoke-sub-ecd3ffc3-r4m9v\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:05.908133 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.908101    4519 generic.go:356] "Generic (PLEG): container finished" podID="8f5189a7-0635-46b2-9b7e-f5601e684119" containerID="df1bf362917cf228b6ab008a13022a2352f9c31d492cb025c4108fac2306806d" exitCode=0
Oct 31 16:17:05.908495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.908191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt" event={"ID":"8f5189a7-0635-46b2-9b7e-f5601e684119","Type":"ContainerDied","Data":"df1bf362917cf228b6ab008a13022a2352f9c31d492cb025c4108fac2306806d"}
Oct 31 16:17:05.922279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:05.922256    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:06.293558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:06.293522    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"]
Oct 31 16:17:06.301520 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:06.301491    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d769e43_791b_4d5c_b7ef_c1b168246c71.slice/crio-c5073e51739a666e56171d2a3df5ad8b3ddd075a0b090140f25d396ce5c268c0 WatchSource:0}: Error finding container c5073e51739a666e56171d2a3df5ad8b3ddd075a0b090140f25d396ce5c268c0: Status 404 returned error can't find the container with id c5073e51739a666e56171d2a3df5ad8b3ddd075a0b090140f25d396ce5c268c0
Oct 31 16:17:06.915582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:06.915499    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v" event={"ID":"9d769e43-791b-4d5c-b7ef-c1b168246c71","Type":"ContainerStarted","Data":"2a38a5e62185bd222af09dd37c4f8a6b5c7caf21726365f3d7b0214e30e798e6"}
Oct 31 16:17:06.915582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:06.915556    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v" event={"ID":"9d769e43-791b-4d5c-b7ef-c1b168246c71","Type":"ContainerStarted","Data":"c5073e51739a666e56171d2a3df5ad8b3ddd075a0b090140f25d396ce5c268c0"}
Oct 31 16:17:07.027797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.027749    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v" podStartSLOduration=2.027735958 podStartE2EDuration="2.027735958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:07.022966642 +0000 UTC m=+870.722895955" watchObservedRunningTime="2026-10-31 16:17:07.027735958 +0000 UTC m=+870.727665302"
Oct 31 16:17:07.058568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.058544    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:07.150833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.150801    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca\") pod \"8f5189a7-0635-46b2-9b7e-f5601e684119\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") "
Oct 31 16:17:07.151018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.150890    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca\") pod \"8f5189a7-0635-46b2-9b7e-f5601e684119\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") "
Oct 31 16:17:07.151093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.151057    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb\" (UniqueName: \"kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb\") pod \"8f5189a7-0635-46b2-9b7e-f5601e684119\" (UID: \"8f5189a7-0635-46b2-9b7e-f5601e684119\") "
Oct 31 16:17:07.151174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.151146    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca" pod "8f5189a7-0635-46b2-9b7e-f5601e684119" (UID: "8f5189a7-0635-46b2-9b7e-f5601e684119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:07.151445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.151428    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:07.152939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.152915    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb" pod "8f5189a7-0635-46b2-9b7e-f5601e684119" (UID: "8f5189a7-0635-46b2-9b7e-f5601e684119"). InnerVolumeSpecName "kube-api-access-4x8wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:07.153011 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.152922    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca" pod "8f5189a7-0635-46b2-9b7e-f5601e684119" (UID: "8f5189a7-0635-46b2-9b7e-f5601e684119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:07.252744 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.252712    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8wb\" (UniqueName: \"kubernetes.io/projected/8f5189a7-0635-46b2-9b7e-f5601e684119-kube-api-access-4x8wb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:07.252744 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.252741    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5189a7-0635-46b2-9b7e-f5601e684119-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:07.922161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.922134    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"
Oct 31 16:17:07.922623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.922135    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt" event={"ID":"8f5189a7-0635-46b2-9b7e-f5601e684119","Type":"ContainerDied","Data":"4cd86c39a29f045f0176bb5463dc7539c375d4f335fda2f86b9358e150facd4e"}
Oct 31 16:17:07.922623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.922276    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cd86c39a29f045f0176bb5463dc7539c375d4f335fda2f86b9358e150facd4e"
Oct 31 16:17:07.923657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.923636    4519 generic.go:356] "Generic (PLEG): container finished" podID="9d769e43-791b-4d5c-b7ef-c1b168246c71" containerID="2a38a5e62185bd222af09dd37c4f8a6b5c7caf21726365f3d7b0214e30e798e6" exitCode=0
Oct 31 16:17:07.923766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:07.923700    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v" event={"ID":"9d769e43-791b-4d5c-b7ef-c1b168246c71","Type":"ContainerDied","Data":"2a38a5e62185bd222af09dd37c4f8a6b5c7caf21726365f3d7b0214e30e798e6"}
Oct 31 16:17:09.075923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.075899    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:09.173510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.173477    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca\") pod \"9d769e43-791b-4d5c-b7ef-c1b168246c71\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") "
Oct 31 16:17:09.173698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.173643    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca\") pod \"9d769e43-791b-4d5c-b7ef-c1b168246c71\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") "
Oct 31 16:17:09.173778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.173712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n\" (UniqueName: \"kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n\") pod \"9d769e43-791b-4d5c-b7ef-c1b168246c71\" (UID: \"9d769e43-791b-4d5c-b7ef-c1b168246c71\") "
Oct 31 16:17:09.173902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.173877    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca" pod "9d769e43-791b-4d5c-b7ef-c1b168246c71" (UID: "9d769e43-791b-4d5c-b7ef-c1b168246c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:09.174089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.174072    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:09.175780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.175752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca" pod "9d769e43-791b-4d5c-b7ef-c1b168246c71" (UID: "9d769e43-791b-4d5c-b7ef-c1b168246c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:09.175780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.175763    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n" pod "9d769e43-791b-4d5c-b7ef-c1b168246c71" (UID: "9d769e43-791b-4d5c-b7ef-c1b168246c71"). InnerVolumeSpecName "kube-api-access-c8n9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:09.275209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.275177    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d769e43-791b-4d5c-b7ef-c1b168246c71-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:09.275209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.275206    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8n9n\" (UniqueName: \"kubernetes.io/projected/9d769e43-791b-4d5c-b7ef-c1b168246c71-kube-api-access-c8n9n\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:09.937019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.936984    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v" event={"ID":"9d769e43-791b-4d5c-b7ef-c1b168246c71","Type":"ContainerDied","Data":"c5073e51739a666e56171d2a3df5ad8b3ddd075a0b090140f25d396ce5c268c0"}
Oct 31 16:17:09.937019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.937003    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"
Oct 31 16:17:09.937200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:09.937057    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5073e51739a666e56171d2a3df5ad8b3ddd075a0b090140f25d396ce5c268c0"
Oct 31 16:17:14.487958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.487924    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"]
Oct 31 16:17:14.488440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.488426    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8f5189a7-0635-46b2-9b7e-f5601e684119" containerName="revoke-keys"
Oct 31 16:17:14.488519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.488441    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5189a7-0635-46b2-9b7e-f5601e684119" containerName="revoke-keys"
Oct 31 16:17:14.488519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.488468    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9d769e43-791b-4d5c-b7ef-c1b168246c71" containerName="revoke-keys"
Oct 31 16:17:14.488519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.488475    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d769e43-791b-4d5c-b7ef-c1b168246c71" containerName="revoke-keys"
Oct 31 16:17:14.488608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.488565    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5189a7-0635-46b2-9b7e-f5601e684119" containerName="revoke-keys"
Oct 31 16:17:14.488608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.488577    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d769e43-791b-4d5c-b7ef-c1b168246c71" containerName="revoke-keys"
Oct 31 16:17:14.491933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.491916    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.544210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.544180    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"]
Oct 31 16:17:14.587726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.587696    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:17:14.587880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.587740    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:17:14.587880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.587759    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:17:14.589063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.589046    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:17:14.632772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.632726    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.632923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.632812    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.632923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.632916    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qzz\" (UniqueName: \"kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.734582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.734523    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.734788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.734607    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.734788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.734694    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qzz\" (UniqueName: \"kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.735125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.735100    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.736836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.736812    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.788850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.788776    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qzz\" (UniqueName: \"kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz\") pod \"maas-api-revoke-sub-c5367cde-2wq9q\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:14.805498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:14.805450    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:15.174751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:15.174668    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"]
Oct 31 16:17:15.180087 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:15.180056    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3fcc06_6cb5_411e_bdac_9e27b2e0c67e.slice/crio-4da9e8a490b1db0929a4785483b59640b1d85a1046928d1c34956f68a778de55 WatchSource:0}: Error finding container 4da9e8a490b1db0929a4785483b59640b1d85a1046928d1c34956f68a778de55: Status 404 returned error can't find the container with id 4da9e8a490b1db0929a4785483b59640b1d85a1046928d1c34956f68a778de55
Oct 31 16:17:15.974659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:15.974623    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q" event={"ID":"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e","Type":"ContainerStarted","Data":"99b1a920b5adf22dbc3e2409829ee1fa43d5b29730bd3b34678211b5239ee39c"}
Oct 31 16:17:15.975051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:15.974684    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q" event={"ID":"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e","Type":"ContainerStarted","Data":"4da9e8a490b1db0929a4785483b59640b1d85a1046928d1c34956f68a778de55"}
Oct 31 16:17:16.085253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:16.085206    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q" podStartSLOduration=2.085193843 podStartE2EDuration="2.085193843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:16.08229692 +0000 UTC m=+879.782226233" watchObservedRunningTime="2026-10-31 16:17:16.085193843 +0000 UTC m=+879.785123154"
Oct 31 16:17:16.982385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:16.982350    4519 generic.go:356] "Generic (PLEG): container finished" podID="3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" containerID="99b1a920b5adf22dbc3e2409829ee1fa43d5b29730bd3b34678211b5239ee39c" exitCode=0
Oct 31 16:17:16.982802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:16.982439    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q" event={"ID":"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e","Type":"ContainerDied","Data":"99b1a920b5adf22dbc3e2409829ee1fa43d5b29730bd3b34678211b5239ee39c"}
Oct 31 16:17:18.125980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.125958    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:18.274670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.274595    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca\") pod \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") "
Oct 31 16:17:18.274670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.274652    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz\" (UniqueName: \"kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz\") pod \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") "
Oct 31 16:17:18.274865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.274690    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca\") pod \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\" (UID: \"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e\") "
Oct 31 16:17:18.275103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.275081    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca" pod "3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" (UID: "3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:18.276632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.276612    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca" pod "3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" (UID: "3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:18.276768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.276752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz" pod "3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" (UID: "3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e"). InnerVolumeSpecName "kube-api-access-q8qzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:18.376200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.376165    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:18.376200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.376194    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8qzz\" (UniqueName: \"kubernetes.io/projected/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-kube-api-access-q8qzz\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:18.376200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.376205    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:18.996646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.996611    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q" event={"ID":"3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e","Type":"ContainerDied","Data":"4da9e8a490b1db0929a4785483b59640b1d85a1046928d1c34956f68a778de55"}
Oct 31 16:17:18.996813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.996659    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da9e8a490b1db0929a4785483b59640b1d85a1046928d1c34956f68a778de55"
Oct 31 16:17:18.996813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:18.996623    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"
Oct 31 16:17:19.906641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:19.906593    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"]
Oct 31 16:17:19.907178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:19.906997    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy" containerID="cri-o://90c934b0e67fc5b7511aa1e6eb5c2686b362b5bb376e2e8939c9871ecb53e0a8" gracePeriod=30
Oct 31 16:17:24.004689 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:17:24.004643    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:17:25.034180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.034141    4519 generic.go:356] "Generic (PLEG): container finished" podID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerID="90c934b0e67fc5b7511aa1e6eb5c2686b362b5bb376e2e8939c9871ecb53e0a8" exitCode=0
Oct 31 16:17:25.034521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.034206    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" event={"ID":"58f4b5fd-e92c-4b54-9797-a3bafa886577","Type":"ContainerDied","Data":"90c934b0e67fc5b7511aa1e6eb5c2686b362b5bb376e2e8939c9871ecb53e0a8"}
Oct 31 16:17:25.172825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.172755    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:17:25.250655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250613    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.250655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250659    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.250899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250679    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.250899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250732    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.250899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250771    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.250899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250799    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.250899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250841    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.251147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250933    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.251147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250952    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.251147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250978    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs\") pod \"58f4b5fd-e92c-4b54-9797-a3bafa886577\" (UID: \"58f4b5fd-e92c-4b54-9797-a3bafa886577\") "
Oct 31 16:17:25.251320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.250959    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.251320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251098    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:25.251320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251168    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.251320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251244    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.251536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251524    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.251586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251548    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.251586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251562    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4b5fd-e92c-4b54-9797-a3bafa886577-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.251586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251580    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.251691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.251594    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.253244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.253220    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "kube-api-access-bdrbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:25.253350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.253290    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:17:25.253437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.253410    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:25.253437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.253429    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy" pod "58f4b5fd-e92c-4b54-9797-a3bafa886577" (UID: "58f4b5fd-e92c-4b54-9797-a3bafa886577"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.353302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.353250    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdrbg\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-kube-api-access-bdrbg\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.353302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.353301    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.353540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.353319    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.353540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:25.353334    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4b5fd-e92c-4b54-9797-a3bafa886577-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:26.041156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:26.041125    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw" event={"ID":"58f4b5fd-e92c-4b54-9797-a3bafa886577","Type":"ContainerDied","Data":"8de1b6b9f14759fbd6a70f2242e4d86d6c552dbfb310876b2f5e065045d202e2"}
Oct 31 16:17:26.041156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:26.041150    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"
Oct 31 16:17:26.041676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:26.041181    4519 scope.go:122] "RemoveContainer" containerID="90c934b0e67fc5b7511aa1e6eb5c2686b362b5bb376e2e8939c9871ecb53e0a8"
Oct 31 16:17:26.244240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:26.244204    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"]
Oct 31 16:17:26.291224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:26.291147    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4lhhw"]
Oct 31 16:17:26.438416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:26.438382    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" path="/var/lib/kubelet/pods/58f4b5fd-e92c-4b54-9797-a3bafa886577/volumes"
Oct 31 16:17:32.840609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.840574    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"]
Oct 31 16:17:32.841107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.841094    4519 cpu_manager.go:414] "containerMap: removing container" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy"
Oct 31 16:17:32.841107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.841107    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy"
Oct 31 16:17:32.841207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.841119    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" containerName="revoke-keys"
Oct 31 16:17:32.841207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.841125    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" containerName="revoke-keys"
Oct 31 16:17:32.841207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.841194    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f4b5fd-e92c-4b54-9797-a3bafa886577" containerName="istio-proxy"
Oct 31 16:17:32.841207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.841204    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" containerName="revoke-keys"
Oct 31 16:17:32.846143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.846122    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:32.897134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.897103    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"]
Oct 31 16:17:32.939253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.939215    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:32.939447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.939422    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:32.939554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.939534    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgbp\" (UniqueName: \"kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:32.940746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.940723    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:17:32.940859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.940792    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:17:32.940859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.940811    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:17:32.942655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:32.942637    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:17:33.040979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.040928    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.040979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.040987    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgbp\" (UniqueName: \"kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.041259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.041168    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.041725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.041704    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.043586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.043564    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.094963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.094898    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgbp\" (UniqueName: \"kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp\") pod \"maas-api-revoke-sub-8d85db3d-vlwfd\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.159983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.159950    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:33.529152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:33.529111    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"]
Oct 31 16:17:33.534579 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:33.534549    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaefea16_9a99_446b_9d95_920bc23c5580.slice/crio-60cb7f93f9975670c96730c238c22f02232e79afd7cde05a893c08b71b27f3c6 WatchSource:0}: Error finding container 60cb7f93f9975670c96730c238c22f02232e79afd7cde05a893c08b71b27f3c6: Status 404 returned error can't find the container with id 60cb7f93f9975670c96730c238c22f02232e79afd7cde05a893c08b71b27f3c6
Oct 31 16:17:34.090397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:34.090358    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd" event={"ID":"caefea16-9a99-446b-9d95-920bc23c5580","Type":"ContainerStarted","Data":"ebeb7673e80a37c0dded6bdee899dc1d0c7585935ed09431a49aefe4991c3ba6"}
Oct 31 16:17:34.090764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:34.090426    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd" event={"ID":"caefea16-9a99-446b-9d95-920bc23c5580","Type":"ContainerStarted","Data":"60cb7f93f9975670c96730c238c22f02232e79afd7cde05a893c08b71b27f3c6"}
Oct 31 16:17:34.198302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:34.198205    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd" podStartSLOduration=2.198192437 podStartE2EDuration="2.198192437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:34.197087414 +0000 UTC m=+897.897016725" watchObservedRunningTime="2026-10-31 16:17:34.198192437 +0000 UTC m=+897.898121755"
Oct 31 16:17:35.098843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:35.098805    4519 generic.go:356] "Generic (PLEG): container finished" podID="caefea16-9a99-446b-9d95-920bc23c5580" containerID="ebeb7673e80a37c0dded6bdee899dc1d0c7585935ed09431a49aefe4991c3ba6" exitCode=0
Oct 31 16:17:35.099241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:35.098889    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd" event={"ID":"caefea16-9a99-446b-9d95-920bc23c5580","Type":"ContainerDied","Data":"ebeb7673e80a37c0dded6bdee899dc1d0c7585935ed09431a49aefe4991c3ba6"}
Oct 31 16:17:36.205389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.205347    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"]
Oct 31 16:17:36.210008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.209982    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.259011 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.258989    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:36.276350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.276323    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdkf\" (UniqueName: \"kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.276519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.276363    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.276596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.276562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.291584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.291556    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"]
Oct 31 16:17:36.338374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.338342    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"]
Oct 31 16:17:36.338869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.338855    4519 cpu_manager.go:414] "containerMap: removing container" podUID="caefea16-9a99-446b-9d95-920bc23c5580" containerName="revoke-keys"
Oct 31 16:17:36.338918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.338871    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caefea16-9a99-446b-9d95-920bc23c5580" containerName="revoke-keys"
Oct 31 16:17:36.338995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.338986    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="caefea16-9a99-446b-9d95-920bc23c5580" containerName="revoke-keys"
Oct 31 16:17:36.342255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.342238    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.377492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377417    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca\") pod \"caefea16-9a99-446b-9d95-920bc23c5580\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") "
Oct 31 16:17:36.377633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca\") pod \"caefea16-9a99-446b-9d95-920bc23c5580\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") "
Oct 31 16:17:36.377712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377701    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp\" (UniqueName: \"kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp\") pod \"caefea16-9a99-446b-9d95-920bc23c5580\" (UID: \"caefea16-9a99-446b-9d95-920bc23c5580\") "
Oct 31 16:17:36.377757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377728    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca" pod "caefea16-9a99-446b-9d95-920bc23c5580" (UID: "caefea16-9a99-446b-9d95-920bc23c5580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:36.377825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdkf\" (UniqueName: \"kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.377860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.377938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.377921    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.378103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.378088    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:36.378622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.378599    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.379838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.379816    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca" pod "caefea16-9a99-446b-9d95-920bc23c5580" (UID: "caefea16-9a99-446b-9d95-920bc23c5580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:36.379925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.379910    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp" pod "caefea16-9a99-446b-9d95-920bc23c5580" (UID: "caefea16-9a99-446b-9d95-920bc23c5580"). InnerVolumeSpecName "kube-api-access-wqgbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:36.380148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.380127    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.386597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.386572    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"]
Oct 31 16:17:36.431564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.431534    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdkf\" (UniqueName: \"kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf\") pod \"maas-api-revoke-sub-d2959b67-6n8gs\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.479198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.479160    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.479399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.479380    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9lx\" (UniqueName: \"kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.479488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.479416    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.479536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.479507    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgbp\" (UniqueName: \"kubernetes.io/projected/caefea16-9a99-446b-9d95-920bc23c5580-kube-api-access-wqgbp\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:36.479536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.479520    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caefea16-9a99-446b-9d95-920bc23c5580-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:36.524209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.524173    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:36.585474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.582988    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9lx\" (UniqueName: \"kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.585474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.583073    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.585474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.583102    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.585474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.584344    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.587885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.587835    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.637183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.637107    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9lx\" (UniqueName: \"kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx\") pod \"maas-api-revoke-sub-6aa900bc-45n8b\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.655347 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.655317    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:36.902602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:36.902531    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"]
Oct 31 16:17:36.907548 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:36.907512    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578aec0c_f3a9_471c_b7b8_98c35fb3bab3.slice/crio-a126cd1e81c212d28d70863399079961899efe5fb7126aec206d715ecda78914 WatchSource:0}: Error finding container a126cd1e81c212d28d70863399079961899efe5fb7126aec206d715ecda78914: Status 404 returned error can't find the container with id a126cd1e81c212d28d70863399079961899efe5fb7126aec206d715ecda78914
Oct 31 16:17:37.031877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:37.031843    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"]
Oct 31 16:17:37.037597 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:37.037573    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b4ad60_92dc_4084_8fe5_d047b6f3aa89.slice/crio-99028f2598c595f5e2936f0260fb16a719f4c58c22969a666bf05705d48b1c1b WatchSource:0}: Error finding container 99028f2598c595f5e2936f0260fb16a719f4c58c22969a666bf05705d48b1c1b: Status 404 returned error can't find the container with id 99028f2598c595f5e2936f0260fb16a719f4c58c22969a666bf05705d48b1c1b
Oct 31 16:17:37.121778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:37.121744    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs" event={"ID":"578aec0c-f3a9-471c-b7b8-98c35fb3bab3","Type":"ContainerStarted","Data":"a126cd1e81c212d28d70863399079961899efe5fb7126aec206d715ecda78914"}
Oct 31 16:17:37.123048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:37.123017    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b" event={"ID":"80b4ad60-92dc-4084-8fe5-d047b6f3aa89","Type":"ContainerStarted","Data":"99028f2598c595f5e2936f0260fb16a719f4c58c22969a666bf05705d48b1c1b"}
Oct 31 16:17:37.124488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:37.124450    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"
Oct 31 16:17:37.124570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:37.124489    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd" event={"ID":"caefea16-9a99-446b-9d95-920bc23c5580","Type":"ContainerDied","Data":"60cb7f93f9975670c96730c238c22f02232e79afd7cde05a893c08b71b27f3c6"}
Oct 31 16:17:37.124570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:37.124523    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60cb7f93f9975670c96730c238c22f02232e79afd7cde05a893c08b71b27f3c6"
Oct 31 16:17:38.132748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:38.132717    4519 generic.go:356] "Generic (PLEG): container finished" podID="80b4ad60-92dc-4084-8fe5-d047b6f3aa89" containerID="1d8397dbdb947479875f4c0b4db51d5d0fae1debf97b538dea9ebe300650eb82" exitCode=0
Oct 31 16:17:38.133210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:38.132800    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b" event={"ID":"80b4ad60-92dc-4084-8fe5-d047b6f3aa89","Type":"ContainerDied","Data":"1d8397dbdb947479875f4c0b4db51d5d0fae1debf97b538dea9ebe300650eb82"}
Oct 31 16:17:38.134738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:38.134712    4519 generic.go:356] "Generic (PLEG): container finished" podID="578aec0c-f3a9-471c-b7b8-98c35fb3bab3" containerID="66cb48e4a08189c967873b14346188b03038dc8e03e3b155b5d0c052eef00d66" exitCode=0
Oct 31 16:17:38.134835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:38.134747    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs" event={"ID":"578aec0c-f3a9-471c-b7b8-98c35fb3bab3","Type":"ContainerDied","Data":"66cb48e4a08189c967873b14346188b03038dc8e03e3b155b5d0c052eef00d66"}
Oct 31 16:17:39.275318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.275285    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:39.312012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.311992    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:39.413986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.413895    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca\") pod \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") "
Oct 31 16:17:39.413986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.413941    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca\") pod \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") "
Oct 31 16:17:39.414216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414131    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca\") pod \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") "
Oct 31 16:17:39.414216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414172    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx\" (UniqueName: \"kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx\") pod \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") "
Oct 31 16:17:39.414324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414247    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca\") pod \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\" (UID: \"80b4ad60-92dc-4084-8fe5-d047b6f3aa89\") "
Oct 31 16:17:39.414324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414293    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf\" (UniqueName: \"kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf\") pod \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\" (UID: \"578aec0c-f3a9-471c-b7b8-98c35fb3bab3\") "
Oct 31 16:17:39.414635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414597    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca" pod "578aec0c-f3a9-471c-b7b8-98c35fb3bab3" (UID: "578aec0c-f3a9-471c-b7b8-98c35fb3bab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:39.414760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414740    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:39.415000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.414972    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca" pod "80b4ad60-92dc-4084-8fe5-d047b6f3aa89" (UID: "80b4ad60-92dc-4084-8fe5-d047b6f3aa89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:39.416511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.416448    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca" pod "80b4ad60-92dc-4084-8fe5-d047b6f3aa89" (UID: "80b4ad60-92dc-4084-8fe5-d047b6f3aa89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:39.416896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.416874    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca" pod "578aec0c-f3a9-471c-b7b8-98c35fb3bab3" (UID: "578aec0c-f3a9-471c-b7b8-98c35fb3bab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:39.417135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.417109    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf" pod "578aec0c-f3a9-471c-b7b8-98c35fb3bab3" (UID: "578aec0c-f3a9-471c-b7b8-98c35fb3bab3"). InnerVolumeSpecName "kube-api-access-lwdkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:39.417242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.417201    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx" pod "80b4ad60-92dc-4084-8fe5-d047b6f3aa89" (UID: "80b4ad60-92dc-4084-8fe5-d047b6f3aa89"). InnerVolumeSpecName "kube-api-access-dk9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:39.516477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.516411    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:39.516477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.516448    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:39.516477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.516480    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9lx\" (UniqueName: \"kubernetes.io/projected/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-kube-api-access-dk9lx\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:39.516760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.516496    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b4ad60-92dc-4084-8fe5-d047b6f3aa89-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:39.516760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:39.516511    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwdkf\" (UniqueName: \"kubernetes.io/projected/578aec0c-f3a9-471c-b7b8-98c35fb3bab3-kube-api-access-lwdkf\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:40.149244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.149207    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"
Oct 31 16:17:40.149442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.149207    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs" event={"ID":"578aec0c-f3a9-471c-b7b8-98c35fb3bab3","Type":"ContainerDied","Data":"a126cd1e81c212d28d70863399079961899efe5fb7126aec206d715ecda78914"}
Oct 31 16:17:40.149442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.149346    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a126cd1e81c212d28d70863399079961899efe5fb7126aec206d715ecda78914"
Oct 31 16:17:40.150951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.150931    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"
Oct 31 16:17:40.151049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.150971    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b" event={"ID":"80b4ad60-92dc-4084-8fe5-d047b6f3aa89","Type":"ContainerDied","Data":"99028f2598c595f5e2936f0260fb16a719f4c58c22969a666bf05705d48b1c1b"}
Oct 31 16:17:40.151049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.151016    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99028f2598c595f5e2936f0260fb16a719f4c58c22969a666bf05705d48b1c1b"
Oct 31 16:17:40.871382 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:40.871341    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:17:49.510197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510162    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"]
Oct 31 16:17:49.510720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510705    4519 cpu_manager.go:414] "containerMap: removing container" podUID="578aec0c-f3a9-471c-b7b8-98c35fb3bab3" containerName="revoke-keys"
Oct 31 16:17:49.510773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510721    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578aec0c-f3a9-471c-b7b8-98c35fb3bab3" containerName="revoke-keys"
Oct 31 16:17:49.510773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510754    4519 cpu_manager.go:414] "containerMap: removing container" podUID="80b4ad60-92dc-4084-8fe5-d047b6f3aa89" containerName="revoke-keys"
Oct 31 16:17:49.510773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510759    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b4ad60-92dc-4084-8fe5-d047b6f3aa89" containerName="revoke-keys"
Oct 31 16:17:49.510867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510840    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b4ad60-92dc-4084-8fe5-d047b6f3aa89" containerName="revoke-keys"
Oct 31 16:17:49.510867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.510853    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="578aec0c-f3a9-471c-b7b8-98c35fb3bab3" containerName="revoke-keys"
Oct 31 16:17:49.515303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.515280    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.612354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.612323    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:17:49.612354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.612324    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:17:49.612354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.612323    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:17:49.612623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.612328    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:17:49.615251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.615233    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"]
Oct 31 16:17:49.627499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.627475    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.627656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.627637    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.627699 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.627665    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgzf\" (UniqueName: \"kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.728415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.728380    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.728611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.728566    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.728611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.728602    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgzf\" (UniqueName: \"kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.729167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.729145    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.730841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.730818    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.782165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.782107    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgzf\" (UniqueName: \"kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf\") pod \"maas-api-revoke-sub-ee10422f-gvc6g\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:49.828415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:49.828391    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:50.195895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:50.195821    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"]
Oct 31 16:17:50.200893 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:50.200866    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f6092f_7ee0_4bb7_8fe2_b81c4a79d7d6.slice/crio-88fd34a96c017903ec1db48bc65285c1ffde16caa4647edcc75b8fdc18c1f748 WatchSource:0}: Error finding container 88fd34a96c017903ec1db48bc65285c1ffde16caa4647edcc75b8fdc18c1f748: Status 404 returned error can't find the container with id 88fd34a96c017903ec1db48bc65285c1ffde16caa4647edcc75b8fdc18c1f748
Oct 31 16:17:50.225278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:50.225243    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g" event={"ID":"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6","Type":"ContainerStarted","Data":"88fd34a96c017903ec1db48bc65285c1ffde16caa4647edcc75b8fdc18c1f748"}
Oct 31 16:17:51.233008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.232977    4519 generic.go:356] "Generic (PLEG): container finished" podID="75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" containerID="8a69d4a51a1f197dfc4e1eb81d7b249a382e348121af0f10259d0da93368c366" exitCode=0
Oct 31 16:17:51.233494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.233024    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g" event={"ID":"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6","Type":"ContainerDied","Data":"8a69d4a51a1f197dfc4e1eb81d7b249a382e348121af0f10259d0da93368c366"}
Oct 31 16:17:51.652176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.652144    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"]
Oct 31 16:17:51.656548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.656523    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.709043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.709001    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"]
Oct 31 16:17:51.852268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.852232    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsl2d\" (UniqueName: \"kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.852449 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.852295    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.852595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.852509    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.953884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.953791    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.954065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.953883    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl2d\" (UniqueName: \"kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.954065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.953944    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.954545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.954517    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:51.956336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:51.956318    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:52.007837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.007808    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsl2d\" (UniqueName: \"kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d\") pod \"maas-api-revoke-sub-67b10338-ttldl\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:52.097056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.097025    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"]
Oct 31 16:17:52.105230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.105203    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.153973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.153946    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"]
Oct 31 16:17:52.256769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.256738    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.257203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.256981    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mbq\" (UniqueName: \"kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.257203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.257030    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.272673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.272633    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:52.358604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.358572    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mbq\" (UniqueName: \"kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.358801 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.358787    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.358963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.358948    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.360200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.360174    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.361439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.361416    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.384038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.384015    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:52.412310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.412286    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mbq\" (UniqueName: \"kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq\") pod \"maas-api-revoke-sub-26a68870-b9gkx\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.419060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.419039    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:52.561433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.561348    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca\") pod \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") "
Oct 31 16:17:52.561433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.561405    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf\" (UniqueName: \"kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf\") pod \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") "
Oct 31 16:17:52.561666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.561526    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca\") pod \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\" (UID: \"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6\") "
Oct 31 16:17:52.561732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.561712    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca" pod "75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" (UID: "75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:52.561923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.561906    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:52.563401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.563378    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca" pod "75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" (UID: "75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:52.563514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.563442    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf" pod "75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" (UID: "75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6"). InnerVolumeSpecName "kube-api-access-mkgzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:52.654076 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.654038    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"]
Oct 31 16:17:52.662499 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:52.659369    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f99a10_a61a_4eac_940f_f2a6f09696fe.slice/crio-3a5a77565049c973802eb531cf6dbe82072c69c42996e81f79dd78021343bc49 WatchSource:0}: Error finding container 3a5a77565049c973802eb531cf6dbe82072c69c42996e81f79dd78021343bc49: Status 404 returned error can't find the container with id 3a5a77565049c973802eb531cf6dbe82072c69c42996e81f79dd78021343bc49
Oct 31 16:17:52.664255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.664229    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkgzf\" (UniqueName: \"kubernetes.io/projected/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-kube-api-access-mkgzf\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:52.664365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.664259    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:52.788264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:52.788230    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"]
Oct 31 16:17:52.793998 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:52.793970    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53ece4b_00fc_4a69_b018_5016eb94676a.slice/crio-208d5750fa55ba7cf41e6c05595db900f942609d035e36df2a4d611b72ac96c0 WatchSource:0}: Error finding container 208d5750fa55ba7cf41e6c05595db900f942609d035e36df2a4d611b72ac96c0: Status 404 returned error can't find the container with id 208d5750fa55ba7cf41e6c05595db900f942609d035e36df2a4d611b72ac96c0
Oct 31 16:17:53.246524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.246490    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g" event={"ID":"75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6","Type":"ContainerDied","Data":"88fd34a96c017903ec1db48bc65285c1ffde16caa4647edcc75b8fdc18c1f748"}
Oct 31 16:17:53.246524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.246508    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"
Oct 31 16:17:53.246793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.246544    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88fd34a96c017903ec1db48bc65285c1ffde16caa4647edcc75b8fdc18c1f748"
Oct 31 16:17:53.248192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.248161    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx" event={"ID":"a53ece4b-00fc-4a69-b018-5016eb94676a","Type":"ContainerStarted","Data":"9853e5f90c02447bcc8253de566199dffca46508b096186299b696553c24b969"}
Oct 31 16:17:53.248311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.248208    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx" event={"ID":"a53ece4b-00fc-4a69-b018-5016eb94676a","Type":"ContainerStarted","Data":"208d5750fa55ba7cf41e6c05595db900f942609d035e36df2a4d611b72ac96c0"}
Oct 31 16:17:53.249887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.249862    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl" event={"ID":"b7f99a10-a61a-4eac-940f-f2a6f09696fe","Type":"ContainerStarted","Data":"4c8beeaad1f43064d9a59a62072726bff326f793c8f77a100ea84b89d1555cb1"}
Oct 31 16:17:53.249985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.249906    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl" event={"ID":"b7f99a10-a61a-4eac-940f-f2a6f09696fe","Type":"ContainerStarted","Data":"3a5a77565049c973802eb531cf6dbe82072c69c42996e81f79dd78021343bc49"}
Oct 31 16:17:53.358910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.358855    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx" podStartSLOduration=1.358841274 podStartE2EDuration="1.358841274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:53.35571438 +0000 UTC m=+917.055643690" watchObservedRunningTime="2026-10-31 16:17:53.358841274 +0000 UTC m=+917.058770640"
Oct 31 16:17:53.513312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:53.513210    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl" podStartSLOduration=2.51319708 podStartE2EDuration="2.51319708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:53.510886592 +0000 UTC m=+917.210815902" watchObservedRunningTime="2026-10-31 16:17:53.51319708 +0000 UTC m=+917.213126429"
Oct 31 16:17:54.003934 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:17:54.003899    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:17:54.014488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.014434    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"]
Oct 31 16:17:54.014992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.014976    4519 cpu_manager.go:414] "containerMap: removing container" podUID="75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" containerName="revoke-keys"
Oct 31 16:17:54.015034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.014994    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" containerName="revoke-keys"
Oct 31 16:17:54.015116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.015099    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" containerName="revoke-keys"
Oct 31 16:17:54.018537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.018519    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.070286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.070257    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"]
Oct 31 16:17:54.076834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.076807    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.076957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.076894    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.077031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.077016    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplb5\" (UniqueName: \"kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.178385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.178342    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.178617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.178510    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplb5\" (UniqueName: \"kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.178617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.178577    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.179160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.179135    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.180770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.180742    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.232133 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.232100    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplb5\" (UniqueName: \"kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5\") pod \"maas-api-revoke-sub-bac69c76-sbr7x\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.257785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.257714    4519 generic.go:356] "Generic (PLEG): container finished" podID="a53ece4b-00fc-4a69-b018-5016eb94676a" containerID="9853e5f90c02447bcc8253de566199dffca46508b096186299b696553c24b969" exitCode=0
Oct 31 16:17:54.257927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.257797    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx" event={"ID":"a53ece4b-00fc-4a69-b018-5016eb94676a","Type":"ContainerDied","Data":"9853e5f90c02447bcc8253de566199dffca46508b096186299b696553c24b969"}
Oct 31 16:17:54.259416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.259392    4519 generic.go:356] "Generic (PLEG): container finished" podID="b7f99a10-a61a-4eac-940f-f2a6f09696fe" containerID="4c8beeaad1f43064d9a59a62072726bff326f793c8f77a100ea84b89d1555cb1" exitCode=0
Oct 31 16:17:54.259526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.259480    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl" event={"ID":"b7f99a10-a61a-4eac-940f-f2a6f09696fe","Type":"ContainerDied","Data":"4c8beeaad1f43064d9a59a62072726bff326f793c8f77a100ea84b89d1555cb1"}
Oct 31 16:17:54.332142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.332117    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:54.712794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:54.712766    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"]
Oct 31 16:17:54.718384 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:54.718347    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122a1b46_dcb6_40b8_82f7_7554fd55779c.slice/crio-e0c3bb73cbbb6c82fa042007eab4f15a5562ac883d8903412e8f72f5f64d5c41 WatchSource:0}: Error finding container e0c3bb73cbbb6c82fa042007eab4f15a5562ac883d8903412e8f72f5f64d5c41: Status 404 returned error can't find the container with id e0c3bb73cbbb6c82fa042007eab4f15a5562ac883d8903412e8f72f5f64d5c41
Oct 31 16:17:55.266886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.266797    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x" event={"ID":"122a1b46-dcb6-40b8-82f7-7554fd55779c","Type":"ContainerStarted","Data":"d384b5c902b2ba9ef0fad66910287ce6ae7f0bb7214f79d9e15eef2a03920f26"}
Oct 31 16:17:55.266886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.266856    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x" event={"ID":"122a1b46-dcb6-40b8-82f7-7554fd55779c","Type":"ContainerStarted","Data":"e0c3bb73cbbb6c82fa042007eab4f15a5562ac883d8903412e8f72f5f64d5c41"}
Oct 31 16:17:55.375270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.375211    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x" podStartSLOduration=2.375192881 podStartE2EDuration="2.375192881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:55.373449502 +0000 UTC m=+919.073378814" watchObservedRunningTime="2026-10-31 16:17:55.375192881 +0000 UTC m=+919.075122192"
Oct 31 16:17:55.479335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.479300    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:55.482350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.482330    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:55.492847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.492828    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca\") pod \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") "
Oct 31 16:17:55.492943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.492855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca\") pod \"a53ece4b-00fc-4a69-b018-5016eb94676a\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") "
Oct 31 16:17:55.492943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.492897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca\") pod \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") "
Oct 31 16:17:55.493061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.492978    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca\") pod \"a53ece4b-00fc-4a69-b018-5016eb94676a\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") "
Oct 31 16:17:55.493132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.493057    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq\" (UniqueName: \"kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq\") pod \"a53ece4b-00fc-4a69-b018-5016eb94676a\" (UID: \"a53ece4b-00fc-4a69-b018-5016eb94676a\") "
Oct 31 16:17:55.493132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.493095    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d\" (UniqueName: \"kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d\") pod \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\" (UID: \"b7f99a10-a61a-4eac-940f-f2a6f09696fe\") "
Oct 31 16:17:55.493585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.493557    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca" pod "b7f99a10-a61a-4eac-940f-f2a6f09696fe" (UID: "b7f99a10-a61a-4eac-940f-f2a6f09696fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:55.493688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.493615    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca" pod "a53ece4b-00fc-4a69-b018-5016eb94676a" (UID: "a53ece4b-00fc-4a69-b018-5016eb94676a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:55.495066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.495044    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.495277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.495071    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.495641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.495615    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca" pod "b7f99a10-a61a-4eac-940f-f2a6f09696fe" (UID: "b7f99a10-a61a-4eac-940f-f2a6f09696fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:55.495852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.495831    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca" pod "a53ece4b-00fc-4a69-b018-5016eb94676a" (UID: "a53ece4b-00fc-4a69-b018-5016eb94676a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:55.496705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.496678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq" pod "a53ece4b-00fc-4a69-b018-5016eb94676a" (UID: "a53ece4b-00fc-4a69-b018-5016eb94676a"). InnerVolumeSpecName "kube-api-access-z5mbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:55.497292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.497271    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d" pod "b7f99a10-a61a-4eac-940f-f2a6f09696fe" (UID: "b7f99a10-a61a-4eac-940f-f2a6f09696fe"). InnerVolumeSpecName "kube-api-access-rsl2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:55.595961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.595924    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mbq\" (UniqueName: \"kubernetes.io/projected/a53ece4b-00fc-4a69-b018-5016eb94676a-kube-api-access-z5mbq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.595961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.595960    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsl2d\" (UniqueName: \"kubernetes.io/projected/b7f99a10-a61a-4eac-940f-f2a6f09696fe-kube-api-access-rsl2d\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.596192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.595978    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7f99a10-a61a-4eac-940f-f2a6f09696fe-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.596192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:55.595992    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53ece4b-00fc-4a69-b018-5016eb94676a-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:56.275122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.275089    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"
Oct 31 16:17:56.275122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.275109    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx" event={"ID":"a53ece4b-00fc-4a69-b018-5016eb94676a","Type":"ContainerDied","Data":"208d5750fa55ba7cf41e6c05595db900f942609d035e36df2a4d611b72ac96c0"}
Oct 31 16:17:56.275633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.275161    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208d5750fa55ba7cf41e6c05595db900f942609d035e36df2a4d611b72ac96c0"
Oct 31 16:17:56.276615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.276597    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"
Oct 31 16:17:56.276750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.276636    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl" event={"ID":"b7f99a10-a61a-4eac-940f-f2a6f09696fe","Type":"ContainerDied","Data":"3a5a77565049c973802eb531cf6dbe82072c69c42996e81f79dd78021343bc49"}
Oct 31 16:17:56.276750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.276683    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a5a77565049c973802eb531cf6dbe82072c69c42996e81f79dd78021343bc49"
Oct 31 16:17:56.278325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.278301    4519 generic.go:356] "Generic (PLEG): container finished" podID="122a1b46-dcb6-40b8-82f7-7554fd55779c" containerID="d384b5c902b2ba9ef0fad66910287ce6ae7f0bb7214f79d9e15eef2a03920f26" exitCode=0
Oct 31 16:17:56.278420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:56.278386    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x" event={"ID":"122a1b46-dcb6-40b8-82f7-7554fd55779c","Type":"ContainerDied","Data":"d384b5c902b2ba9ef0fad66910287ce6ae7f0bb7214f79d9e15eef2a03920f26"}
Oct 31 16:17:57.427092 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.427069    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:57.519747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.518561    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca\") pod \"122a1b46-dcb6-40b8-82f7-7554fd55779c\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") "
Oct 31 16:17:57.519747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.518665    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca\") pod \"122a1b46-dcb6-40b8-82f7-7554fd55779c\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") "
Oct 31 16:17:57.519747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.518737    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5\" (UniqueName: \"kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5\") pod \"122a1b46-dcb6-40b8-82f7-7554fd55779c\" (UID: \"122a1b46-dcb6-40b8-82f7-7554fd55779c\") "
Oct 31 16:17:57.522574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.522344    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca" pod "122a1b46-dcb6-40b8-82f7-7554fd55779c" (UID: "122a1b46-dcb6-40b8-82f7-7554fd55779c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:57.522574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.522434    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5" pod "122a1b46-dcb6-40b8-82f7-7554fd55779c" (UID: "122a1b46-dcb6-40b8-82f7-7554fd55779c"). InnerVolumeSpecName "kube-api-access-nplb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:57.526543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.526515    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca" pod "122a1b46-dcb6-40b8-82f7-7554fd55779c" (UID: "122a1b46-dcb6-40b8-82f7-7554fd55779c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:57.620706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.620632    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:57.620706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.620660    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a1b46-dcb6-40b8-82f7-7554fd55779c-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:57.620706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:57.620670    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nplb5\" (UniqueName: \"kubernetes.io/projected/122a1b46-dcb6-40b8-82f7-7554fd55779c-kube-api-access-nplb5\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:58.292578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.292541    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x" event={"ID":"122a1b46-dcb6-40b8-82f7-7554fd55779c","Type":"ContainerDied","Data":"e0c3bb73cbbb6c82fa042007eab4f15a5562ac883d8903412e8f72f5f64d5c41"}
Oct 31 16:17:58.292748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.292616    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c3bb73cbbb6c82fa042007eab4f15a5562ac883d8903412e8f72f5f64d5c41"
Oct 31 16:17:58.292748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.292554    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"
Oct 31 16:17:58.443223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443191    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"]
Oct 31 16:17:58.443808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443791    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b7f99a10-a61a-4eac-940f-f2a6f09696fe" containerName="revoke-keys"
Oct 31 16:17:58.443853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443810    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f99a10-a61a-4eac-940f-f2a6f09696fe" containerName="revoke-keys"
Oct 31 16:17:58.443853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443835    4519 cpu_manager.go:414] "containerMap: removing container" podUID="122a1b46-dcb6-40b8-82f7-7554fd55779c" containerName="revoke-keys"
Oct 31 16:17:58.443853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443840    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122a1b46-dcb6-40b8-82f7-7554fd55779c" containerName="revoke-keys"
Oct 31 16:17:58.443943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443864    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a53ece4b-00fc-4a69-b018-5016eb94676a" containerName="revoke-keys"
Oct 31 16:17:58.443943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443870    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53ece4b-00fc-4a69-b018-5016eb94676a" containerName="revoke-keys"
Oct 31 16:17:58.444001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443950    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f99a10-a61a-4eac-940f-f2a6f09696fe" containerName="revoke-keys"
Oct 31 16:17:58.444001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443963    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="122a1b46-dcb6-40b8-82f7-7554fd55779c" containerName="revoke-keys"
Oct 31 16:17:58.444001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.443972    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53ece4b-00fc-4a69-b018-5016eb94676a" containerName="revoke-keys"
Oct 31 16:17:58.447549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.447533    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.497374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.497338    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"distinct-w2-e7f13a-kserve-self-signed-certs\""
Oct 31 16:17:58.511643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.511617    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"]
Oct 31 16:17:58.531163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.531137    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.531310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.531196    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.531310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.531275    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.531310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.531304    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhs2\" (UniqueName: \"kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.531430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.531406    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.531482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.531442    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.632927    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.632984    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633054    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633109    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633145    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633177    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhs2\" (UniqueName: \"kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633404    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633425    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.633598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.633496    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.635381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.635356    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.635604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.635586    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.687402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.687369    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhs2\" (UniqueName: \"kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2\") pod \"distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:58.759823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:58.759791    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:17:59.151837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:59.151795    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"]
Oct 31 16:17:59.155994 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:17:59.155969    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7364ac2_fb65_43b4_87a6_1bca2d5b28cd.slice/crio-8189105e135fc0ae3264829e0d1d64f2555abc582b8f3386dad27af52fd08d6d WatchSource:0}: Error finding container 8189105e135fc0ae3264829e0d1d64f2555abc582b8f3386dad27af52fd08d6d: Status 404 returned error can't find the container with id 8189105e135fc0ae3264829e0d1d64f2555abc582b8f3386dad27af52fd08d6d
Oct 31 16:17:59.301042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:59.301005    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" event={"ID":"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd","Type":"ContainerStarted","Data":"274281c410757ec16eaa5392fdb2ddb8cb06f8123dbdcba5d576d047f71c0a3e"}
Oct 31 16:17:59.301187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:59.301054    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" event={"ID":"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd","Type":"ContainerStarted","Data":"8189105e135fc0ae3264829e0d1d64f2555abc582b8f3386dad27af52fd08d6d"}
Oct 31 16:17:59.411520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:59.411406    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podStartSLOduration=1.411392411 podStartE2EDuration="1.411392411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:59.407824209 +0000 UTC m=+923.107753521" watchObservedRunningTime="2026-10-31 16:17:59.411392411 +0000 UTC m=+923.111321721"
Oct 31 16:17:59.759893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:17:59.759856    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:18:02.894149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.894099    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"]
Oct 31 16:18:02.898509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.898487    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:02.949945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.949916    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"]
Oct 31 16:18:02.993363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.993327    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:18:02.993541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.993407    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:18:02.994988 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.994968    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:18:02.995216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:02.995201    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:18:03.000977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.000944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twcn\" (UniqueName: \"kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.001081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.000987    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.001175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.001127    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.101939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.101892    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twcn\" (UniqueName: \"kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.101939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.101940    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.102205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.102011    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.102627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.102606    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.104277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.104260    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.126692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.126665    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"]
Oct 31 16:18:03.130835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.130818    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.155744 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.155689    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twcn\" (UniqueName: \"kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn\") pod \"maas-api-revoke-sub-eb7bedb1-56ch6\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.183636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.183611    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"]
Oct 31 16:18:03.202964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.202936    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrm7\" (UniqueName: \"kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.203091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.202977    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.203091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.203002    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.212267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.212246    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:03.295168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.295138    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"]
Oct 31 16:18:03.300386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.300364    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.304790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.304762    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrm7\" (UniqueName: \"kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.304934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.304825    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.304934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.304869    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.305748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.305725    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.309291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.309242    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.350913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.350820    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f7821a-openshift-default-dockercfg-pbwpd\""
Oct 31 16:18:03.354429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.354396    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"]
Oct 31 16:18:03.359645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.359621    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrm7\" (UniqueName: \"kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7\") pod \"maas-api-revoke-sub-cd41ca27-tnqzk\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.406450 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406382    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406450 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406419    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406492    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406632    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406755    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406819    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406906    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.406975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406962    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvscv\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.407012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.406992    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.444905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.444873    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:03.507925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.507881    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508052 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.507988    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508098    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508183    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvscv\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508221    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508296    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.508439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.508320    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.510350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.510318    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.511074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.511049    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.511300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.511283    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.512040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.512016    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.512180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.512153    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.512264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.512176    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.514305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.514284    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.564022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.563996    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.564905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.564871    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvscv\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.585281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.585257    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"]
Oct 31 16:18:03.590976 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:03.590949    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60877a1b_0839_4e9b_8c13_348fc4f46f25.slice/crio-a99a68706f3b8f34905bba3ab99336bb120604e755ebffe8aae748caa5399215 WatchSource:0}: Error finding container a99a68706f3b8f34905bba3ab99336bb120604e755ebffe8aae748caa5399215: Status 404 returned error can't find the container with id a99a68706f3b8f34905bba3ab99336bb120604e755ebffe8aae748caa5399215
Oct 31 16:18:03.626990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.626967    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:03.818005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:03.817974    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"]
Oct 31 16:18:03.822941 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:03.822912    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17c3e5a_843d_4c85_b16f_13701843e1c9.slice/crio-399091bf2d9ff7982c8a4f72bb5f12095571d978268466957f3ef540860dd39b WatchSource:0}: Error finding container 399091bf2d9ff7982c8a4f72bb5f12095571d978268466957f3ef540860dd39b: Status 404 returned error can't find the container with id 399091bf2d9ff7982c8a4f72bb5f12095571d978268466957f3ef540860dd39b
Oct 31 16:18:04.005032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.005003    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"]
Oct 31 16:18:04.009331 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:04.009300    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e3ea94_9d0a_486f_a60a_29e4ff843650.slice/crio-ecb57578d694ad32b67ab143ebcef4354db467a5a51b58e0d0c391c700bf7d9b WatchSource:0}: Error finding container ecb57578d694ad32b67ab143ebcef4354db467a5a51b58e0d0c391c700bf7d9b: Status 404 returned error can't find the container with id ecb57578d694ad32b67ab143ebcef4354db467a5a51b58e0d0c391c700bf7d9b
Oct 31 16:18:04.011792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.011760    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:04.011880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.011826    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:04.011880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.011859    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:04.348251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.348162    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk" event={"ID":"e17c3e5a-843d-4c85-b16f-13701843e1c9","Type":"ContainerStarted","Data":"b1836f7318077312072f24c4a7222c7bdc71d05c09ec59d8fc1842cc1a72e263"}
Oct 31 16:18:04.348251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.348230    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk" event={"ID":"e17c3e5a-843d-4c85-b16f-13701843e1c9","Type":"ContainerStarted","Data":"399091bf2d9ff7982c8a4f72bb5f12095571d978268466957f3ef540860dd39b"}
Oct 31 16:18:04.350003 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.349977    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6" event={"ID":"60877a1b-0839-4e9b-8c13-348fc4f46f25","Type":"ContainerStarted","Data":"30d6b371da8eb61a1036c703bcc9ff577fa3c1ca2d7effe01edd18f3a42ae86d"}
Oct 31 16:18:04.350131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.350028    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6" event={"ID":"60877a1b-0839-4e9b-8c13-348fc4f46f25","Type":"ContainerStarted","Data":"a99a68706f3b8f34905bba3ab99336bb120604e755ebffe8aae748caa5399215"}
Oct 31 16:18:04.351677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.351656    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" event={"ID":"b4e3ea94-9d0a-486f-a60a-29e4ff843650","Type":"ContainerStarted","Data":"dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a"}
Oct 31 16:18:04.351781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.351706    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" event={"ID":"b4e3ea94-9d0a-486f-a60a-29e4ff843650","Type":"ContainerStarted","Data":"ecb57578d694ad32b67ab143ebcef4354db467a5a51b58e0d0c391c700bf7d9b"}
Oct 31 16:18:04.457404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.457349    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk" podStartSLOduration=1.457336097 podStartE2EDuration="1.457336097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:04.454913027 +0000 UTC m=+928.154842338" watchObservedRunningTime="2026-10-31 16:18:04.457336097 +0000 UTC m=+928.157265446"
Oct 31 16:18:04.592520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.584322    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6" podStartSLOduration=2.584301032 podStartE2EDuration="2.584301032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:04.565480783 +0000 UTC m=+928.265410091" watchObservedRunningTime="2026-10-31 16:18:04.584301032 +0000 UTC m=+928.284230345"
Oct 31 16:18:04.625662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.625589    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"]
Oct 31 16:18:04.629689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.629671    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:04.629808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.629752    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.693174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.693142    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ef6d93-openshift-default-dockercfg-dzsjw\""
Oct 31 16:18:04.700832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.700775    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podStartSLOduration=1.700756695 podStartE2EDuration="1.700756695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:04.693489205 +0000 UTC m=+928.393418517" watchObservedRunningTime="2026-10-31 16:18:04.700756695 +0000 UTC m=+928.400686006"
Oct 31 16:18:04.706038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.706012    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"]
Oct 31 16:18:04.723945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.723912    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724067    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724143    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724209    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvfv\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724286    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724339    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.724411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.724372    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825567    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825606    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825624    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825675    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825704    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvfv\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825738    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825794    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.825849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825825    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.826212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.825903    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.826212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.826152    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.826400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.826378    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.826670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.826641    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.826786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.826671    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.826786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.826731    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.828164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.828144    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.828752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.828729    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.893406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.893320    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.893614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.893411    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvfv\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.942684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.942652    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:04.948060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.948035    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"]
Oct 31 16:18:04.953032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:04.953012    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.028749    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.028825    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.028915    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.029057    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.029103    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.029146    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqggg\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.029211    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.029295    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.029362    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.031344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.030757    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"]
Oct 31 16:18:05.131078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131047    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131085    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131313 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131240    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131424    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131521    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131586    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131698    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131762 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131734    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131762 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131749    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqggg\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131778    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131800    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.131855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.131847    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.132378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.132352    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.132648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.132625    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.134213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.134191    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.134582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.134562    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.193497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.193389    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.194548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.194265    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqggg\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dgfgj\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.290479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.289702    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:05.353153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.352065    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"]
Oct 31 16:18:05.395497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.394518    4519 generic.go:356] "Generic (PLEG): container finished" podID="e17c3e5a-843d-4c85-b16f-13701843e1c9" containerID="b1836f7318077312072f24c4a7222c7bdc71d05c09ec59d8fc1842cc1a72e263" exitCode=0
Oct 31 16:18:05.395497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.394617    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk" event={"ID":"e17c3e5a-843d-4c85-b16f-13701843e1c9","Type":"ContainerDied","Data":"b1836f7318077312072f24c4a7222c7bdc71d05c09ec59d8fc1842cc1a72e263"}
Oct 31 16:18:05.397098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.397058    4519 generic.go:356] "Generic (PLEG): container finished" podID="60877a1b-0839-4e9b-8c13-348fc4f46f25" containerID="30d6b371da8eb61a1036c703bcc9ff577fa3c1ca2d7effe01edd18f3a42ae86d" exitCode=0
Oct 31 16:18:05.397623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.397602    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6" event={"ID":"60877a1b-0839-4e9b-8c13-348fc4f46f25","Type":"ContainerDied","Data":"30d6b371da8eb61a1036c703bcc9ff577fa3c1ca2d7effe01edd18f3a42ae86d"}
Oct 31 16:18:05.398233 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:05.398207    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8ee24f_7d6e_445f_88f4_08bbf07da809.slice/crio-c7d98e5cfd1512a6210b166039af73720a44eb8b9945f93960de064385122015 WatchSource:0}: Error finding container c7d98e5cfd1512a6210b166039af73720a44eb8b9945f93960de064385122015: Status 404 returned error can't find the container with id c7d98e5cfd1512a6210b166039af73720a44eb8b9945f93960de064385122015
Oct 31 16:18:05.400738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.400706    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.400834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.400768    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.400834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.400815    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.629141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.629107    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:18:05.629284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.629172    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 16:18:05.682482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.665602    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"]
Oct 31 16:18:05.706712 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:05.706626    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e200262_0853_4f53_b7af_561ba9cfab51.slice/crio-99fa5299a0c9587641d816c914e975c9f3771e16ccd969a7d7e2e0e323a46c83 WatchSource:0}: Error finding container 99fa5299a0c9587641d816c914e975c9f3771e16ccd969a7d7e2e0e323a46c83: Status 404 returned error can't find the container with id 99fa5299a0c9587641d816c914e975c9f3771e16ccd969a7d7e2e0e323a46c83
Oct 31 16:18:05.709588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.709051    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.709588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.709142    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.709588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:05.709189    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:06.444651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.444617    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" event={"ID":"4b8ee24f-7d6e-445f-88f4-08bbf07da809","Type":"ContainerStarted","Data":"87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010"}
Oct 31 16:18:06.445123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.444661    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" event={"ID":"4b8ee24f-7d6e-445f-88f4-08bbf07da809","Type":"ContainerStarted","Data":"c7d98e5cfd1512a6210b166039af73720a44eb8b9945f93960de064385122015"}
Oct 31 16:18:06.445123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.444680    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" event={"ID":"0e200262-0853-4f53-b7af-561ba9cfab51","Type":"ContainerStarted","Data":"bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc"}
Oct 31 16:18:06.445123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.444695    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" event={"ID":"0e200262-0853-4f53-b7af-561ba9cfab51","Type":"ContainerStarted","Data":"99fa5299a0c9587641d816c914e975c9f3771e16ccd969a7d7e2e0e323a46c83"}
Oct 31 16:18:06.545482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.523471    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"]
Oct 31 16:18:06.665046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.662071    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:18:06.665046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.662129    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 16:18:06.726188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.726156    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"]
Oct 31 16:18:06.726549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.726481    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.806499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.765850    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.806499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.765919    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.806499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.765973    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzls\" (UniqueName: \"kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.881520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.867027    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzls\" (UniqueName: \"kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.881520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.867227    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.881520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.867276    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.891479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.883041    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.999886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.956620    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:06.999886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:06.957741    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:07.124175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.051082    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:18:07.124175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.051144    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 16:18:07.124175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.052114    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzls\" (UniqueName: \"kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls\") pod \"maas-api-revoke-sub-e594afa1-9v7v4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:07.124175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.062648    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podStartSLOduration=3.062628415 podStartE2EDuration="3.062628415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:07.052604816 +0000 UTC m=+930.752534127" watchObservedRunningTime="2026-10-31 16:18:07.062628415 +0000 UTC m=+930.762557727"
Oct 31 16:18:07.124175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.063833    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podStartSLOduration=3.063816728 podStartE2EDuration="3.063816728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:06.858761136 +0000 UTC m=+930.558690447" watchObservedRunningTime="2026-10-31 16:18:07.063816728 +0000 UTC m=+930.763746041"
Oct 31 16:18:07.394481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.313767    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:07.394481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.319326    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:07.394481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.319834    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:07.394481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.319895    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:07.663712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.663684    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:07.664398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.664377    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk" event={"ID":"e17c3e5a-843d-4c85-b16f-13701843e1c9","Type":"ContainerDied","Data":"399091bf2d9ff7982c8a4f72bb5f12095571d978268466957f3ef540860dd39b"}
Oct 31 16:18:07.664578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.664562    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="399091bf2d9ff7982c8a4f72bb5f12095571d978268466957f3ef540860dd39b"
Oct 31 16:18:07.665657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.665630    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:18:07.669476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.665808    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 16:18:07.679476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.676249    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6" event={"ID":"60877a1b-0839-4e9b-8c13-348fc4f46f25","Type":"ContainerDied","Data":"a99a68706f3b8f34905bba3ab99336bb120604e755ebffe8aae748caa5399215"}
Oct 31 16:18:07.697621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.697591    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99a68706f3b8f34905bba3ab99336bb120604e755ebffe8aae748caa5399215"
Oct 31 16:18:07.697621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.697505    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:07.864545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.864474    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca\") pod \"60877a1b-0839-4e9b-8c13-348fc4f46f25\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") "
Oct 31 16:18:07.865033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.865015    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca\") pod \"e17c3e5a-843d-4c85-b16f-13701843e1c9\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") "
Oct 31 16:18:07.865173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.865158    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca\") pod \"e17c3e5a-843d-4c85-b16f-13701843e1c9\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") "
Oct 31 16:18:07.865303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.865286    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7\" (UniqueName: \"kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7\") pod \"e17c3e5a-843d-4c85-b16f-13701843e1c9\" (UID: \"e17c3e5a-843d-4c85-b16f-13701843e1c9\") "
Oct 31 16:18:07.869479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.865443    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca\") pod \"60877a1b-0839-4e9b-8c13-348fc4f46f25\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") "
Oct 31 16:18:07.869830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.869807    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn\" (UniqueName: \"kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn\") pod \"60877a1b-0839-4e9b-8c13-348fc4f46f25\" (UID: \"60877a1b-0839-4e9b-8c13-348fc4f46f25\") "
Oct 31 16:18:07.891479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.865584    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca" pod "e17c3e5a-843d-4c85-b16f-13701843e1c9" (UID: "e17c3e5a-843d-4c85-b16f-13701843e1c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:07.891689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.889211    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca" pod "60877a1b-0839-4e9b-8c13-348fc4f46f25" (UID: "60877a1b-0839-4e9b-8c13-348fc4f46f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:07.961001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.960650    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:18:07.961001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.960691    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn" pod "60877a1b-0839-4e9b-8c13-348fc4f46f25" (UID: "60877a1b-0839-4e9b-8c13-348fc4f46f25"). InnerVolumeSpecName "kube-api-access-8twcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:07.961001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.960727    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 16:18:07.961001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.960769    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca" pod "60877a1b-0839-4e9b-8c13-348fc4f46f25" (UID: "60877a1b-0839-4e9b-8c13-348fc4f46f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:07.961001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.960652    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca" pod "e17c3e5a-843d-4c85-b16f-13701843e1c9" (UID: "e17c3e5a-843d-4c85-b16f-13701843e1c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:07.963855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.961736    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7" pod "e17c3e5a-843d-4c85-b16f-13701843e1c9" (UID: "e17c3e5a-843d-4c85-b16f-13701843e1c9"). InnerVolumeSpecName "kube-api-access-bcrm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:07.994502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.987370    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:07.994502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.987409    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twcn\" (UniqueName: \"kubernetes.io/projected/60877a1b-0839-4e9b-8c13-348fc4f46f25-kube-api-access-8twcn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:07.994502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.987426    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60877a1b-0839-4e9b-8c13-348fc4f46f25-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:07.994502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.987443    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:07.994502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.987475    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e17c3e5a-843d-4c85-b16f-13701843e1c9-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:07.994502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:07.987490    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrm7\" (UniqueName: \"kubernetes.io/projected/e17c3e5a-843d-4c85-b16f-13701843e1c9-kube-api-access-bcrm7\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:08.304582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.297620    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:08.304582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.297719    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:08.547260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.547174    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"]
Oct 31 16:18:08.643481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.627296    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:18:08.643481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.627350    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 16:18:08.748482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.746126    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"
Oct 31 16:18:08.748482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.746793    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4" event={"ID":"b4e0e1db-8ebe-40e6-8188-63f1498e49a4","Type":"ContainerStarted","Data":"039b584b6f97f10e3390b67715db392cd2e65a3bfd0b0aa63a61dffbdb634bba"}
Oct 31 16:18:08.748482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.748073    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"
Oct 31 16:18:08.760956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.760923    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:18:08.847600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.847502    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:18:08.943477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.943090    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:18:08.943477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:08.943145    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 16:18:09.295305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.294857    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:09.295305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.294922    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:09.631185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.631152    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:18:09.631488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.631443    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 16:18:09.756142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.756105    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4" event={"ID":"b4e0e1db-8ebe-40e6-8188-63f1498e49a4","Type":"ContainerStarted","Data":"cab01d6dce518aae6ddcbd7ccdafbc533c71b12308717c9115983ace2ba41583"}
Oct 31 16:18:09.834438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.834411    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:18:09.892534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.880549    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4" podStartSLOduration=3.8805319430000003 podStartE2EDuration="3.880531943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:09.877891219 +0000 UTC m=+933.577820533" watchObservedRunningTime="2026-10-31 16:18:09.880531943 +0000 UTC m=+933.580461257"
Oct 31 16:18:09.946009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.945951    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:18:09.946211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.946007    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"]
Oct 31 16:18:09.946211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.946062    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 16:18:09.948549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.948528    4519 cpu_manager.go:414] "containerMap: removing container" podUID="60877a1b-0839-4e9b-8c13-348fc4f46f25" containerName="revoke-keys"
Oct 31 16:18:09.948727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.948712    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60877a1b-0839-4e9b-8c13-348fc4f46f25" containerName="revoke-keys"
Oct 31 16:18:09.948857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.948846    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e17c3e5a-843d-4c85-b16f-13701843e1c9" containerName="revoke-keys"
Oct 31 16:18:09.948953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.948942    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17c3e5a-843d-4c85-b16f-13701843e1c9" containerName="revoke-keys"
Oct 31 16:18:09.949271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.949244    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="60877a1b-0839-4e9b-8c13-348fc4f46f25" containerName="revoke-keys"
Oct 31 16:18:09.949403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.949392    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17c3e5a-843d-4c85-b16f-13701843e1c9" containerName="revoke-keys"
Oct 31 16:18:09.953818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:09.953805    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.011497 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:18:09.974021    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e0e1db_8ebe_40e6_8188_63f1498e49a4.slice/crio-conmon-cab01d6dce518aae6ddcbd7ccdafbc533c71b12308717c9115983ace2ba41583.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:18:10.039722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.039678    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"]
Oct 31 16:18:10.139569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.139531    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k984f\" (UniqueName: \"kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.139891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.139851    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.140207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.140176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.241949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.241653    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k984f\" (UniqueName: \"kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.241949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.241743    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.241949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.241928    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.243883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.243854    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.245990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.245968    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.297477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.296878    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:10.297477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.296941    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:10.323477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.320938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k984f\" (UniqueName: \"kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f\") pod \"maas-api-revoke-sub-2b54f894-rrljb\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.598538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.595263    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:10.629006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.628818    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:18:10.629006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.628874    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 16:18:10.829506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.814343    4519 generic.go:356] "Generic (PLEG): container finished" podID="b4e0e1db-8ebe-40e6-8188-63f1498e49a4" containerID="cab01d6dce518aae6ddcbd7ccdafbc533c71b12308717c9115983ace2ba41583" exitCode=0
Oct 31 16:18:10.829506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.820515    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4" event={"ID":"b4e0e1db-8ebe-40e6-8188-63f1498e49a4","Type":"ContainerDied","Data":"cab01d6dce518aae6ddcbd7ccdafbc533c71b12308717c9115983ace2ba41583"}
Oct 31 16:18:10.946142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.943546    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:18:10.946142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:10.943609    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 16:18:11.065098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.065055    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"]
Oct 31 16:18:11.117995 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:11.117945    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08c7e70_d9d7_4283_8194_8b855d3030ae.slice/crio-c9248609e7c21679b0ae220639c1240c9bf9136e95727cc26ddce447458348a7 WatchSource:0}: Error finding container c9248609e7c21679b0ae220639c1240c9bf9136e95727cc26ddce447458348a7: Status 404 returned error can't find the container with id c9248609e7c21679b0ae220639c1240c9bf9136e95727cc26ddce447458348a7
Oct 31 16:18:11.294249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.294210    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:11.294573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.294543    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:11.631753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.631676    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:11.632607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.632589    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:11.633096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.633076    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:18:11.823446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.823391    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb" event={"ID":"c08c7e70-d9d7-4283-8194-8b855d3030ae","Type":"ContainerStarted","Data":"22632dbdec7783f9c32a71a0062b080b0b8012295a7bb68c68eb5dd143c8be13"}
Oct 31 16:18:11.823633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.823496    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb" event={"ID":"c08c7e70-d9d7-4283-8194-8b855d3030ae","Type":"ContainerStarted","Data":"c9248609e7c21679b0ae220639c1240c9bf9136e95727cc26ddce447458348a7"}
Oct 31 16:18:11.945472 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.943787    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:18:11.945472 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.943845    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 16:18:12.003484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:11.968297    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb" podStartSLOduration=2.968276649 podStartE2EDuration="2.968276649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:11.964893248 +0000 UTC m=+935.664822558" watchObservedRunningTime="2026-10-31 16:18:11.968276649 +0000 UTC m=+935.668205961"
Oct 31 16:18:12.042537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.042513    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:12.106294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.106264    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca\") pod \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") "
Oct 31 16:18:12.106520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.106433    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls\" (UniqueName: \"kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls\") pod \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") "
Oct 31 16:18:12.106520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.106494    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca\") pod \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\" (UID: \"b4e0e1db-8ebe-40e6-8188-63f1498e49a4\") "
Oct 31 16:18:12.107124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.107099    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca" pod "b4e0e1db-8ebe-40e6-8188-63f1498e49a4" (UID: "b4e0e1db-8ebe-40e6-8188-63f1498e49a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:12.108768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.108730    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca" pod "b4e0e1db-8ebe-40e6-8188-63f1498e49a4" (UID: "b4e0e1db-8ebe-40e6-8188-63f1498e49a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:12.108948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.108931    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls" pod "b4e0e1db-8ebe-40e6-8188-63f1498e49a4" (UID: "b4e0e1db-8ebe-40e6-8188-63f1498e49a4"). InnerVolumeSpecName "kube-api-access-bhzls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:12.207738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.207664    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.207738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.207690    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzls\" (UniqueName: \"kubernetes.io/projected/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-kube-api-access-bhzls\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.207738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.207704    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0e1db-8ebe-40e6-8188-63f1498e49a4-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.291776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.291739    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:12.291944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.291809    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:12.832333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.832293    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"
Oct 31 16:18:12.832532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.832298    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4" event={"ID":"b4e0e1db-8ebe-40e6-8188-63f1498e49a4","Type":"ContainerDied","Data":"039b584b6f97f10e3390b67715db392cd2e65a3bfd0b0aa63a61dffbdb634bba"}
Oct 31 16:18:12.832532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.832437    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="039b584b6f97f10e3390b67715db392cd2e65a3bfd0b0aa63a61dffbdb634bba"
Oct 31 16:18:12.834129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.834104    4519 generic.go:356] "Generic (PLEG): container finished" podID="c08c7e70-d9d7-4283-8194-8b855d3030ae" containerID="22632dbdec7783f9c32a71a0062b080b0b8012295a7bb68c68eb5dd143c8be13" exitCode=0
Oct 31 16:18:12.834252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.834177    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb" event={"ID":"c08c7e70-d9d7-4283-8194-8b855d3030ae","Type":"ContainerDied","Data":"22632dbdec7783f9c32a71a0062b080b0b8012295a7bb68c68eb5dd143c8be13"}
Oct 31 16:18:12.949593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.949560    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:12.950169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.949961    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:12.951500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:12.950740    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:18:13.290799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:13.290757    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dgfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:18:13.290991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:13.290827    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 16:18:13.991094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:13.991064    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:14.027090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.027059    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f\" (UniqueName: \"kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f\") pod \"c08c7e70-d9d7-4283-8194-8b855d3030ae\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") "
Oct 31 16:18:14.027238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.027166    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca\") pod \"c08c7e70-d9d7-4283-8194-8b855d3030ae\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") "
Oct 31 16:18:14.027238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.027216    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca\") pod \"c08c7e70-d9d7-4283-8194-8b855d3030ae\" (UID: \"c08c7e70-d9d7-4283-8194-8b855d3030ae\") "
Oct 31 16:18:14.027657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.027625    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca" pod "c08c7e70-d9d7-4283-8194-8b855d3030ae" (UID: "c08c7e70-d9d7-4283-8194-8b855d3030ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:14.027859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.027837    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:14.029244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.029214    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca" pod "c08c7e70-d9d7-4283-8194-8b855d3030ae" (UID: "c08c7e70-d9d7-4283-8194-8b855d3030ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:14.029347 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.029324    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f" pod "c08c7e70-d9d7-4283-8194-8b855d3030ae" (UID: "c08c7e70-d9d7-4283-8194-8b855d3030ae"). InnerVolumeSpecName "kube-api-access-k984f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:14.128592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.128510    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k984f\" (UniqueName: \"kubernetes.io/projected/c08c7e70-d9d7-4283-8194-8b855d3030ae-kube-api-access-k984f\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:14.128592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.128538    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08c7e70-d9d7-4283-8194-8b855d3030ae-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:14.546124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.545989    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:14.546694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.546669    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:14.547909 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.547890    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:18:14.851441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.851399    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"
Oct 31 16:18:14.851441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.851417    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb" event={"ID":"c08c7e70-d9d7-4283-8194-8b855d3030ae","Type":"ContainerDied","Data":"c9248609e7c21679b0ae220639c1240c9bf9136e95727cc26ddce447458348a7"}
Oct 31 16:18:14.851712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:14.851497    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9248609e7c21679b0ae220639c1240c9bf9136e95727cc26ddce447458348a7"
Oct 31 16:18:15.437294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:15.437260    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.288145    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"]
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.289006    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c08c7e70-d9d7-4283-8194-8b855d3030ae" containerName="revoke-keys"
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.289025    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08c7e70-d9d7-4283-8194-8b855d3030ae" containerName="revoke-keys"
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.289045    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b4e0e1db-8ebe-40e6-8188-63f1498e49a4" containerName="revoke-keys"
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.289054    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e0e1db-8ebe-40e6-8188-63f1498e49a4" containerName="revoke-keys"
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.289225    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08c7e70-d9d7-4283-8194-8b855d3030ae" containerName="revoke-keys"
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.289238    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e0e1db-8ebe-40e6-8188-63f1498e49a4" containerName="revoke-keys"
Oct 31 16:18:17.298477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.293666    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.384731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.384505    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.385156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.385129    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.385325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.385309    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnwb\" (UniqueName: \"kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.391803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.391771    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:18:17.391915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.391874    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:18:17.391988 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.391968    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:18:17.579624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.413619    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"]
Oct 31 16:18:17.629479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.602915    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:18:17.629479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.609311    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.629479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.609385    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.629479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.609433    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnwb\" (UniqueName: \"kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.665481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.631910    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.743037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.633410    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.743037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.683903    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnwb\" (UniqueName: \"kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb\") pod \"maas-api-revoke-sub-00359b39-sgz7m\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:17.929890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:17.929813    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:18.309012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:18.308973    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"]
Oct 31 16:18:18.313541 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:18.313511    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373bd2d7_edc3_4bac_9d65_53d901f46d09.slice/crio-dcb45f96adf13cd6de0850175be5f4ef5970ea2cdc02aaca23f7242fe139d3b3 WatchSource:0}: Error finding container dcb45f96adf13cd6de0850175be5f4ef5970ea2cdc02aaca23f7242fe139d3b3: Status 404 returned error can't find the container with id dcb45f96adf13cd6de0850175be5f4ef5970ea2cdc02aaca23f7242fe139d3b3
Oct 31 16:18:18.887337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:18.887231    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m" event={"ID":"373bd2d7-edc3-4bac-9d65-53d901f46d09","Type":"ContainerStarted","Data":"c62ca0eb34d3e58085cff86a971a35f9308c3f2a5ccd126c57ba2e934fb7e63c"}
Oct 31 16:18:18.887337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:18.887296    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m" event={"ID":"373bd2d7-edc3-4bac-9d65-53d901f46d09","Type":"ContainerStarted","Data":"dcb45f96adf13cd6de0850175be5f4ef5970ea2cdc02aaca23f7242fe139d3b3"}
Oct 31 16:18:18.999824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:18.999772    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m" podStartSLOduration=1.999758921 podStartE2EDuration="1.999758921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:18.998819111 +0000 UTC m=+942.698748423" watchObservedRunningTime="2026-10-31 16:18:18.999758921 +0000 UTC m=+942.699688231"
Oct 31 16:18:19.898907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:19.898870    4519 generic.go:356] "Generic (PLEG): container finished" podID="373bd2d7-edc3-4bac-9d65-53d901f46d09" containerID="c62ca0eb34d3e58085cff86a971a35f9308c3f2a5ccd126c57ba2e934fb7e63c" exitCode=0
Oct 31 16:18:19.899393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:19.898950    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m" event={"ID":"373bd2d7-edc3-4bac-9d65-53d901f46d09","Type":"ContainerDied","Data":"c62ca0eb34d3e58085cff86a971a35f9308c3f2a5ccd126c57ba2e934fb7e63c"}
Oct 31 16:18:20.002345 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.002307    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"]
Oct 31 16:18:20.006481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.006448    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.059744 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.059710    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"]
Oct 31 16:18:20.071670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.071636    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.071911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.071881    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.071949    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.071977    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrnk\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.072020    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.072081    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.072111    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.072151    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.072386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.072266    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.173740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173660    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.173894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173805    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.173894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173835    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.173894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173865    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.173894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173883    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrnk\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173957    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.173978    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.174010    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.174078    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.174165    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.174423    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.174506    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.174759 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.174741    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.176149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.176125    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.176572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.176540    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.228360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.228328    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.228541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.228394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrnk\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fqh2n\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.320785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.320753    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:20.746700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.746666    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"]
Oct 31 16:18:20.750859 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:20.750834    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e9db4a_ba99_4082_bc38_037581fe4449.slice/crio-68684bca0755a4fb224f8776465fd6d1cd98575aff0925955d28c6afe402873e WatchSource:0}: Error finding container 68684bca0755a4fb224f8776465fd6d1cd98575aff0925955d28c6afe402873e: Status 404 returned error can't find the container with id 68684bca0755a4fb224f8776465fd6d1cd98575aff0925955d28c6afe402873e
Oct 31 16:18:20.753388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.753355    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:20.753440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.753422    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:20.753495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.753468    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:20.907945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.907904    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" event={"ID":"88e9db4a-ba99-4082-bc38-037581fe4449","Type":"ContainerStarted","Data":"cd464a24f1a0f9fa29904b5f5795150a00d990dbbe9b114c59324bced752dd23"}
Oct 31 16:18:20.908354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:20.908003    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" event={"ID":"88e9db4a-ba99-4082-bc38-037581fe4449","Type":"ContainerStarted","Data":"68684bca0755a4fb224f8776465fd6d1cd98575aff0925955d28c6afe402873e"}
Oct 31 16:18:21.023567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.023436    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podStartSLOduration=2.023419423 podStartE2EDuration="2.023419423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:21.019538968 +0000 UTC m=+944.719468281" watchObservedRunningTime="2026-10-31 16:18:21.023419423 +0000 UTC m=+944.723348734"
Oct 31 16:18:21.032735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.032710    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:21.084838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.084805    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca\") pod \"373bd2d7-edc3-4bac-9d65-53d901f46d09\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") "
Oct 31 16:18:21.085040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.084937    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca\") pod \"373bd2d7-edc3-4bac-9d65-53d901f46d09\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") "
Oct 31 16:18:21.085120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.085103    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb\" (UniqueName: \"kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb\") pod \"373bd2d7-edc3-4bac-9d65-53d901f46d09\" (UID: \"373bd2d7-edc3-4bac-9d65-53d901f46d09\") "
Oct 31 16:18:21.085323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.085281    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca" pod "373bd2d7-edc3-4bac-9d65-53d901f46d09" (UID: "373bd2d7-edc3-4bac-9d65-53d901f46d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:21.085636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.085612    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:21.087030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.087006    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca" pod "373bd2d7-edc3-4bac-9d65-53d901f46d09" (UID: "373bd2d7-edc3-4bac-9d65-53d901f46d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:21.087137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.087117    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb" pod "373bd2d7-edc3-4bac-9d65-53d901f46d09" (UID: "373bd2d7-edc3-4bac-9d65-53d901f46d09"). InnerVolumeSpecName "kube-api-access-jwnwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:21.186612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.186578    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnwb\" (UniqueName: \"kubernetes.io/projected/373bd2d7-edc3-4bac-9d65-53d901f46d09-kube-api-access-jwnwb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:21.186612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.186610    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373bd2d7-edc3-4bac-9d65-53d901f46d09-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:21.321121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.321039    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:21.322505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.322480    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:21.322649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.322527    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:21.916279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.916246    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"
Oct 31 16:18:21.916683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.916275    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m" event={"ID":"373bd2d7-edc3-4bac-9d65-53d901f46d09","Type":"ContainerDied","Data":"dcb45f96adf13cd6de0850175be5f4ef5970ea2cdc02aaca23f7242fe139d3b3"}
Oct 31 16:18:21.916683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:21.916324    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb45f96adf13cd6de0850175be5f4ef5970ea2cdc02aaca23f7242fe139d3b3"
Oct 31 16:18:22.321158 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:22.321120    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:22.321352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:22.321179    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:23.321713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:23.321677    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:23.322132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:23.321731    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:24.017603 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:18:24.017565    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:18:24.321216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:24.321124    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:24.321216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:24.321183    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:25.321991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.321959    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:25.322422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.322024    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:25.827008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.826969    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"]
Oct 31 16:18:25.827673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.827650    4519 cpu_manager.go:414] "containerMap: removing container" podUID="373bd2d7-edc3-4bac-9d65-53d901f46d09" containerName="revoke-keys"
Oct 31 16:18:25.827673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.827668    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373bd2d7-edc3-4bac-9d65-53d901f46d09" containerName="revoke-keys"
Oct 31 16:18:25.827830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.827750    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="373bd2d7-edc3-4bac-9d65-53d901f46d09" containerName="revoke-keys"
Oct 31 16:18:25.831521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.831501    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:25.897090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.897058    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"]
Oct 31 16:18:25.938257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.938219    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:18:25.938257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.938237    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:18:25.938560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.938237    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:18:25.938560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.938336    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:18:25.943692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.943666    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlm4r\" (UniqueName: \"kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:25.943844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.943707    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:25.943844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:25.943802    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.045145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.045110    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlm4r\" (UniqueName: \"kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.045320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.045154    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.045320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.045200    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.045875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.045849    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.047631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.047609    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.099374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.099297    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlm4r\" (UniqueName: \"kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r\") pod \"maas-api-revoke-sub-a52cba53-444p8\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.145157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.145126    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:26.321167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.321119    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:26.321374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.321172    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:26.548063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.548029    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"]
Oct 31 16:18:26.552511 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:26.552480    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a5785d_612b_4b79_881d_077c9a1db489.slice/crio-ade4abcdd8210e8e4fe9156c4e46dc66a18588e065e095ad73f8897a7254aaf9 WatchSource:0}: Error finding container ade4abcdd8210e8e4fe9156c4e46dc66a18588e065e095ad73f8897a7254aaf9: Status 404 returned error can't find the container with id ade4abcdd8210e8e4fe9156c4e46dc66a18588e065e095ad73f8897a7254aaf9
Oct 31 16:18:26.952759 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:26.952724    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8" event={"ID":"54a5785d-612b-4b79-881d-077c9a1db489","Type":"ContainerStarted","Data":"ade4abcdd8210e8e4fe9156c4e46dc66a18588e065e095ad73f8897a7254aaf9"}
Oct 31 16:18:27.321327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:27.321293    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:27.321530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:27.321355    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:27.962176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:27.962145    4519 generic.go:356] "Generic (PLEG): container finished" podID="54a5785d-612b-4b79-881d-077c9a1db489" containerID="8b418c560fc4c3c261b1f944c4286b375d6b9ae1735ca0a30143bab7a55a0ceb" exitCode=0
Oct 31 16:18:27.962613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:27.962239    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8" event={"ID":"54a5785d-612b-4b79-881d-077c9a1db489","Type":"ContainerDied","Data":"8b418c560fc4c3c261b1f944c4286b375d6b9ae1735ca0a30143bab7a55a0ceb"}
Oct 31 16:18:28.321377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:28.321291    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fqh2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:18:28.321377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:28.321356    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 16:18:29.122128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.122090    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:29.185656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.185625    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r\" (UniqueName: \"kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r\") pod \"54a5785d-612b-4b79-881d-077c9a1db489\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") "
Oct 31 16:18:29.185819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.185693    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca\") pod \"54a5785d-612b-4b79-881d-077c9a1db489\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") "
Oct 31 16:18:29.185819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.185750    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca\") pod \"54a5785d-612b-4b79-881d-077c9a1db489\" (UID: \"54a5785d-612b-4b79-881d-077c9a1db489\") "
Oct 31 16:18:29.186258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.186225    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca" pod "54a5785d-612b-4b79-881d-077c9a1db489" (UID: "54a5785d-612b-4b79-881d-077c9a1db489"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:29.187826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.187802    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r" pod "54a5785d-612b-4b79-881d-077c9a1db489" (UID: "54a5785d-612b-4b79-881d-077c9a1db489"). InnerVolumeSpecName "kube-api-access-dlm4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:29.187911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.187889    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca" pod "54a5785d-612b-4b79-881d-077c9a1db489" (UID: "54a5785d-612b-4b79-881d-077c9a1db489"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:29.287095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.287019    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:29.287095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.287048    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a5785d-612b-4b79-881d-077c9a1db489-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:29.287095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.287060    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlm4r\" (UniqueName: \"kubernetes.io/projected/54a5785d-612b-4b79-881d-077c9a1db489-kube-api-access-dlm4r\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:29.325939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.325909    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:29.326295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.326279    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:29.326996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.326978    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:18:29.982066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.982029    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"
Oct 31 16:18:29.982241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.982065    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8" event={"ID":"54a5785d-612b-4b79-881d-077c9a1db489","Type":"ContainerDied","Data":"ade4abcdd8210e8e4fe9156c4e46dc66a18588e065e095ad73f8897a7254aaf9"}
Oct 31 16:18:29.982241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:29.982109    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade4abcdd8210e8e4fe9156c4e46dc66a18588e065e095ad73f8897a7254aaf9"
Oct 31 16:18:32.949722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:32.949671    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"]
Oct 31 16:18:32.950620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:32.950602    4519 cpu_manager.go:414] "containerMap: removing container" podUID="54a5785d-612b-4b79-881d-077c9a1db489" containerName="revoke-keys"
Oct 31 16:18:32.950663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:32.950622    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a5785d-612b-4b79-881d-077c9a1db489" containerName="revoke-keys"
Oct 31 16:18:32.950838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:32.950820    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a5785d-612b-4b79-881d-077c9a1db489" containerName="revoke-keys"
Oct 31 16:18:32.954835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:32.954816    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.006568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.006531    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"]
Oct 31 16:18:33.028689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.028659    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.028834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.028703    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.028834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.028724    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.028944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.028841    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.029025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.029001    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.029071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.029036    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.029071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.029052    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.029151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.029127    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjq7l\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.029185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.029176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.048545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.048505    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4fe3a4-openshift-default-dockercfg-njqjl\""
Oct 31 16:18:33.129934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.129897    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.129941    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.129961    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130003    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130143    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130169    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130193    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130231    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjq7l\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130267    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130617    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130696    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130715    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130831    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.130935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.130911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.132505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.132482    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.132637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.132622    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.183649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.183618    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.183794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.183724    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjq7l\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.267904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.267868    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:33.696531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.696440    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"]
Oct 31 16:18:33.700599 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:18:33.700575    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b460663_dd16_49f3_bc01_c606f7afe226.slice/crio-561d6bc9574f33180bd313b79fc05e85d6ae9ed19f47bcbf44ee66918af037b6 WatchSource:0}: Error finding container 561d6bc9574f33180bd313b79fc05e85d6ae9ed19f47bcbf44ee66918af037b6: Status 404 returned error can't find the container with id 561d6bc9574f33180bd313b79fc05e85d6ae9ed19f47bcbf44ee66918af037b6
Oct 31 16:18:33.703468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.703423    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:33.703538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.703507    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:33.703572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:33.703539    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:34.013332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:34.013293    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" event={"ID":"2b460663-dd16-49f3-bc01-c606f7afe226","Type":"ContainerStarted","Data":"bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4"}
Oct 31 16:18:34.013771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:34.013368    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" event={"ID":"2b460663-dd16-49f3-bc01-c606f7afe226","Type":"ContainerStarted","Data":"561d6bc9574f33180bd313b79fc05e85d6ae9ed19f47bcbf44ee66918af037b6"}
Oct 31 16:18:34.126324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:34.126269    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podStartSLOduration=2.126253477 podStartE2EDuration="2.126253477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:34.123161606 +0000 UTC m=+957.823090918" watchObservedRunningTime="2026-10-31 16:18:34.126253477 +0000 UTC m=+957.826182787"
Oct 31 16:18:34.268690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:34.268607    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:34.269916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:34.269889    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:34.270046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:34.269933    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:35.276070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:35.276031    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:35.276682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:35.276647    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:36.281480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:36.274859    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:36.281480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:36.274926    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:37.274477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:37.269849    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:37.274477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:37.269914    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:38.275485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:38.274770    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:38.275485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:38.274833    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:39.282481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:39.275893    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:39.282481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:39.275962    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:40.269156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:40.269122    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 16:18:40.269358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:40.269181    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 16:18:41.334370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:41.334338    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:41.334921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:41.334826    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:41.335581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:41.335562    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:18:49.797657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:18:49.797616    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:18:54.004036 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:18:54.003996    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:19:04.100933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.100902    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"]
Oct 31 16:19:04.105302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.105284    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.195982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.195944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.195986    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196017    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196140    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196180    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196248    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196289    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjs7w\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196412    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.196762 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.196728    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.208362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.208335    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"]
Oct 31 16:19:04.297652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297621    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.297824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297665    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.297824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297683    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.297824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297704    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.297824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297737    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.297824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297776    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.297824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297835    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjs7w\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.297931    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.298094    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.298163    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.298194    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.298550    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.298873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.298850    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.300165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.300144    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.300291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.300270    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.351649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.351583    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.351772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.351726    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjs7w\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.418955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.418924    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:04.844407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.844367    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"]
Oct 31 16:19:04.849139 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:19:04.849113    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087a6981_f457_48a6_b9a2_5e57e7ec351a.slice/crio-1b3bc0596167fc39376bed9d54c8f361e50301ad2a93ad90d9a5a969ffabe2f8 WatchSource:0}: Error finding container 1b3bc0596167fc39376bed9d54c8f361e50301ad2a93ad90d9a5a969ffabe2f8: Status 404 returned error can't find the container with id 1b3bc0596167fc39376bed9d54c8f361e50301ad2a93ad90d9a5a969ffabe2f8
Oct 31 16:19:04.851516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.851483    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:04.851593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.851552    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:04.851593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:04.851582    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:05.238930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:05.238889    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" event={"ID":"087a6981-f457-48a6-b9a2-5e57e7ec351a","Type":"ContainerStarted","Data":"8eeaeb604a6b8eb647b71ba87d5f30e07608f5f8654a8f65bfe88eb008b7de01"}
Oct 31 16:19:05.239333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:05.238956    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" event={"ID":"087a6981-f457-48a6-b9a2-5e57e7ec351a","Type":"ContainerStarted","Data":"1b3bc0596167fc39376bed9d54c8f361e50301ad2a93ad90d9a5a969ffabe2f8"}
Oct 31 16:19:05.350931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:05.350867    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podStartSLOduration=1.35084464 podStartE2EDuration="1.35084464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:05.34859527 +0000 UTC m=+989.048524582" watchObservedRunningTime="2026-10-31 16:19:05.35084464 +0000 UTC m=+989.050773952"
Oct 31 16:19:05.419186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:05.419154    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:05.420838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:05.420814    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 16:19:05.420975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:05.420855    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 16:19:06.420978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:06.420942    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 16:19:06.421639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:06.421592    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 16:19:07.419741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.419710    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 16:19:07.419929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.419760    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 16:19:07.450760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.450730    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"]
Oct 31 16:19:07.455124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.455106    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.497690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.497660    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.497890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.497705    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.497890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.497812    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.497890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.497834    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4wl\" (UniqueName: \"kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.497890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.497883    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.498136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.497946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.508784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.508753    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"]
Oct 31 16:19:07.527137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.527109    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"kube-root-ca.crt\""
Oct 31 16:19:07.550196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.550147    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"openshift-service-ca.crt\""
Oct 31 16:19:07.550196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.550165    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"distinct-w1-4fe3a4-kserve-self-signed-certs\""
Oct 31 16:19:07.550196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.550182    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"default-dockercfg-jbp9j\""
Oct 31 16:19:07.603668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.603605    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.604536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.604504    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4wl\" (UniqueName: \"kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.604670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.604598    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.604734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.604716    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.605742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.604844    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.605742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.604924    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.605742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.605359    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.605742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.605648    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.606020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.605776    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.620489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.611351    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.625088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.625059    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.659046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.659015    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4wl\" (UniqueName: \"kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl\") pod \"distinct-w1-4fe3a4-kserve-7998794479-bpt7n\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:07.781491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:07.779697    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:08.227745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:08.227600    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"]
Oct 31 16:19:08.236257 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:19:08.236226    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6761d701_7651_4a9f_b397_0f29df36ce96.slice/crio-a283ced7cc58e6c82d688f719a7c6cfaa38cfab7ddd231746224cd52b5f14f58 WatchSource:0}: Error finding container a283ced7cc58e6c82d688f719a7c6cfaa38cfab7ddd231746224cd52b5f14f58: Status 404 returned error can't find the container with id a283ced7cc58e6c82d688f719a7c6cfaa38cfab7ddd231746224cd52b5f14f58
Oct 31 16:19:08.295243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:08.295211    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" event={"ID":"6761d701-7651-4a9f-b397-0f29df36ce96","Type":"ContainerStarted","Data":"a283ced7cc58e6c82d688f719a7c6cfaa38cfab7ddd231746224cd52b5f14f58"}
Oct 31 16:19:08.419375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:08.419333    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 16:19:08.419576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:08.419389    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 16:19:09.303478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:09.303431    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" event={"ID":"6761d701-7651-4a9f-b397-0f29df36ce96","Type":"ContainerStarted","Data":"1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa"}
Oct 31 16:19:09.415364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:09.415315    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podStartSLOduration=2.415299978 podStartE2EDuration="2.415299978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:09.411814378 +0000 UTC m=+993.111743690" watchObservedRunningTime="2026-10-31 16:19:09.415299978 +0000 UTC m=+993.115229342"
Oct 31 16:19:09.419957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:09.419930    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 16:19:09.420085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:09.419973    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 16:19:09.784477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:09.781544    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:10.419761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:10.419724    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 16:19:10.420241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:10.419784    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 16:19:12.420815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:12.420775    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:19:12.421233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:12.420857    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:19:12.475958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:12.475926    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:12.476243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:12.476220    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:12.477121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:12.477099    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:19:17.780285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:17.780251    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:17.792215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:17.792190    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:18.376711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:18.376682    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:19:24.003954 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:19:24.003914    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:19:29.424991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:29.424941    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:19:35.521106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.521072    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"]
Oct 31 16:19:35.525833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.525811    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.579688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.579653    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"]
Oct 31 16:19:35.579838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.579727    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.579838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.579822    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.579945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.579878    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.580011 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.579996    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.580159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.580140    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ztp8\" (UniqueName: \"kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.580239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.580223    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.620559 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.620534    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"distinct-w5-ef6d93-kserve-self-signed-certs\""
Oct 31 16:19:35.681549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.681822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681582    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.681822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681660    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.681822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681695    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.681822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681738    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.682126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681879    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ztp8\" (UniqueName: \"kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.682126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.681998    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.682280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.682249    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.682470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.682434    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.687732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.687700    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.690995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.690967    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.735850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.735816    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ztp8\" (UniqueName: \"kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8\") pod \"distinct-w5-ef6d93-kserve-67889bcb57-r4x9t\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:35.838079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:35.837985    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:36.230709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:36.230676    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"]
Oct 31 16:19:36.235420 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:19:36.235392    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod044ab7fa_9b54_47a0_990f_4b1fb0e049d1.slice/crio-69694cacaaefe67bee17099e894eef6f99dbfb10c452c09c22cea7be51cf317b WatchSource:0}: Error finding container 69694cacaaefe67bee17099e894eef6f99dbfb10c452c09c22cea7be51cf317b: Status 404 returned error can't find the container with id 69694cacaaefe67bee17099e894eef6f99dbfb10c452c09c22cea7be51cf317b
Oct 31 16:19:36.498123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:36.498035    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" event={"ID":"044ab7fa-9b54-47a0-990f-4b1fb0e049d1","Type":"ContainerStarted","Data":"b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe"}
Oct 31 16:19:36.498123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:36.498076    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" event={"ID":"044ab7fa-9b54-47a0-990f-4b1fb0e049d1","Type":"ContainerStarted","Data":"69694cacaaefe67bee17099e894eef6f99dbfb10c452c09c22cea7be51cf317b"}
Oct 31 16:19:36.607837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:36.607778    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podStartSLOduration=1.6077601339999998 podStartE2EDuration="1.607760134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:36.605571685 +0000 UTC m=+1020.305500994" watchObservedRunningTime="2026-10-31 16:19:36.607760134 +0000 UTC m=+1020.307689446"
Oct 31 16:19:36.838735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:36.838665    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:45.838780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:45.838741    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:45.851266 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:45.851234    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:46.579131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:46.579103    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:19:48.823578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.823478    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"]
Oct 31 16:19:48.828551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.828527    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.881810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.881778    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"]
Oct 31 16:19:48.946544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946511    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dk22\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.946741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946648    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.946741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946690    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.946868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946813    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.946934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946867    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.946934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946883    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.947040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.947040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.946964    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:48.947149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:48.947059    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048158    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048230    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk22\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048329    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048352    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048422    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048495    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048521    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048558    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048588    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048655    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048734    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048874    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.048982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.048937    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.049338 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.049310    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.050943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.050918    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.051033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.050950    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.102126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.102053    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.102126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.102055    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk22\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.143389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.143350    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:49.581335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.581310    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"]
Oct 31 16:19:49.584466 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:19:49.584428    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e574cac_dbd0_4e63_9a50_396cf7bde4fb.slice/crio-95775183eb5aa46f38447f090cb1e94f91b297b73d6063b7f8e03ac5aa502968 WatchSource:0}: Error finding container 95775183eb5aa46f38447f090cb1e94f91b297b73d6063b7f8e03ac5aa502968: Status 404 returned error can't find the container with id 95775183eb5aa46f38447f090cb1e94f91b297b73d6063b7f8e03ac5aa502968
Oct 31 16:19:49.587197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.587163    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.587301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.587231    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.587301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:49.587262    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:50.609112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:50.609076    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" event={"ID":"6e574cac-dbd0-4e63-9a50-396cf7bde4fb","Type":"ContainerStarted","Data":"ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749"}
Oct 31 16:19:50.609580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:50.609133    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" event={"ID":"6e574cac-dbd0-4e63-9a50-396cf7bde4fb","Type":"ContainerStarted","Data":"95775183eb5aa46f38447f090cb1e94f91b297b73d6063b7f8e03ac5aa502968"}
Oct 31 16:19:50.794410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:50.794339    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podStartSLOduration=2.79431859 podStartE2EDuration="2.79431859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:50.765379001 +0000 UTC m=+1034.465308316" watchObservedRunningTime="2026-10-31 16:19:50.79431859 +0000 UTC m=+1034.494247903"
Oct 31 16:19:51.144141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:51.144106    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:51.145575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:51.145549    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 16:19:51.145696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:51.145598    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 16:19:52.144021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:52.143987    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 16:19:52.144439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:52.144041    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 16:19:53.143939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.143904    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 16:19:53.144123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.143955    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 16:19:53.893934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.893895    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"]
Oct 31 16:19:53.898197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.898172    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.934401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.934373    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.934567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.934423    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.934628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.934570    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.934771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.934745    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.934908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.934799    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.934908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.934833    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gvg\" (UniqueName: \"kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:53.939942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.939919    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"]
Oct 31 16:19:53.990587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:53.989749    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"distinct-2-w2-e7f13a-kserve-self-signed-certs\""
Oct 31 16:19:54.017795 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:19:54.017760    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.036540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.036700    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.036748    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.036775    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gvg\" (UniqueName: \"kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.036857    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.036908    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.037508    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.038088    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.038752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.038235    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.040000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.039973    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.046077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.046047    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.100882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.100849    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gvg\" (UniqueName: \"kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg\") pod \"distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") " pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.144064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.143975    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 16:19:54.144064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.144048    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 16:19:54.212753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.212722    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:54.625339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:54.624840    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"]
Oct 31 16:19:54.637834 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:19:54.637796    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod372f17d9_af5d_4e0c_9ac5_25ee9daffa81.slice/crio-6a891654541b2be426d95755df654af4f9eaad78830b8856d84a74cca70ded3c WatchSource:0}: Error finding container 6a891654541b2be426d95755df654af4f9eaad78830b8856d84a74cca70ded3c: Status 404 returned error can't find the container with id 6a891654541b2be426d95755df654af4f9eaad78830b8856d84a74cca70ded3c
Oct 31 16:19:55.144235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:55.144200    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 16:19:55.144641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:55.144264    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 16:19:55.649253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:55.649218    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" event={"ID":"372f17d9-af5d-4e0c-9ac5-25ee9daffa81","Type":"ContainerStarted","Data":"b3b5efc5391874f7566c1359ddfb620d96eac7c6a060a7cda9feb4563c37e406"}
Oct 31 16:19:55.649476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:55.649261    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" event={"ID":"372f17d9-af5d-4e0c-9ac5-25ee9daffa81","Type":"ContainerStarted","Data":"6a891654541b2be426d95755df654af4f9eaad78830b8856d84a74cca70ded3c"}
Oct 31 16:19:55.760432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:55.760380    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podStartSLOduration=2.760365583 podStartE2EDuration="2.760365583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:55.756088657 +0000 UTC m=+1039.456017980" watchObservedRunningTime="2026-10-31 16:19:55.760365583 +0000 UTC m=+1039.460294897"
Oct 31 16:19:56.144552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:56.144520    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 16:19:56.144946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:56.144575    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 16:19:56.213201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:56.213169    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:19:57.148419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:57.148390    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:57.148829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:57.148812    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:19:57.149363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:19:57.149345    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:20:04.213066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:04.213027    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:20:04.226081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:04.226054    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:20:04.736441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:04.736412    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:20:11.557893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:11.557860    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:20:19.935940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.935905    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"]
Oct 31 16:20:19.941404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.941373    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.961891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.961865    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.961912    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962003    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvwt\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962181    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962200    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962264    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962303    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:19.962341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:19.962323    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.030786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.030753    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"]
Oct 31 16:20:20.064042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064057    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064184    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvwt\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064206    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064225    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064258    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064303    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064328    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064371    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064684    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064848 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064827    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.064993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.064843    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.065141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.065120    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.065216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.065191    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.066995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.066973    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.067096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.067043    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.117743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.117717    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.118257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.118237    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvwt\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.257166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.257131    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:20.687828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.687745    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"]
Oct 31 16:20:20.692372 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:20:20.692346    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a759655_221d_4180_af64_e8bf585bcbdb.slice/crio-bc00d393d7084a95f032c946e6fbde7cbde6a0608092364df1c97a0893a0b3eb WatchSource:0}: Error finding container bc00d393d7084a95f032c946e6fbde7cbde6a0608092364df1c97a0893a0b3eb: Status 404 returned error can't find the container with id bc00d393d7084a95f032c946e6fbde7cbde6a0608092364df1c97a0893a0b3eb
Oct 31 16:20:20.695058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.695020    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:20.695144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.695105    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:20.695179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.695147    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:20.841632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.841581    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" event={"ID":"0a759655-221d-4180-af64-e8bf585bcbdb","Type":"ContainerStarted","Data":"15bc41744296e6640d8ba30a2c375696a8d1ef9c3c4423278c864944d0eb99bd"}
Oct 31 16:20:20.841830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.841652    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" event={"ID":"0a759655-221d-4180-af64-e8bf585bcbdb","Type":"ContainerStarted","Data":"bc00d393d7084a95f032c946e6fbde7cbde6a0608092364df1c97a0893a0b3eb"}
Oct 31 16:20:20.958999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:20.958074    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podStartSLOduration=1.958056478 podStartE2EDuration="1.958056478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:20.951110802 +0000 UTC m=+1064.651040124" watchObservedRunningTime="2026-10-31 16:20:20.958056478 +0000 UTC m=+1064.657985792"
Oct 31 16:20:21.257527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:21.257490    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:21.258567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:21.258545    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:21.258682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:21.258601    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:22.258360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:22.258321    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:22.258776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:22.258382    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:23.258325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:23.258288    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:23.258550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:23.258343    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:24.015238 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:20:24.015202    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:20:24.258180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:24.258144    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:24.258371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:24.258203    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:25.257885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:25.257850    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:25.258293 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:25.257917    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:26.257688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:26.257647    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:26.257889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:26.257721    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:27.258395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:27.258359    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:20:27.258837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:27.258425    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 16:20:28.261552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:28.261523    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:28.262029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:28.262003    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:28.262563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:28.262544    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:20:37.089115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.089082    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"]
Oct 31 16:20:37.093520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.093499    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.145143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.145109    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"]
Oct 31 16:20:37.187640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.187613    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"kube-root-ca.crt\""
Oct 31 16:20:37.187809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.187660    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"openshift-service-ca.crt\""
Oct 31 16:20:37.190095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.190070    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"default-dockercfg-zbwr8\""
Oct 31 16:20:37.190095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.190075    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"distinct-2-w4-59357a-kserve-self-signed-certs\""
Oct 31 16:20:37.195894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.195875    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.195967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.195907    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.195967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.195922    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.196118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.196099    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.196168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.196151    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8kb\" (UniqueName: \"kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.196265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.196252    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.297927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.297885    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.297951    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.297989    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.298134    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.298198    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8kb\" (UniqueName: \"kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.298333    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.298632    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.298958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.298945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.299485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.299424    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.300642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.300620    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.301505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.301447    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.356845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.356774    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8kb\" (UniqueName: \"kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb\") pod \"distinct-2-w4-59357a-kserve-769dff4844-m7fl4\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.408594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.408562    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:37.782118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.782081    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"]
Oct 31 16:20:37.786935 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:20:37.786908    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667e8d38_e5ab_4af8_a35a_def8a1083241.slice/crio-22314caefce78ab1167e83b8e1851d48851d68792d9b931eb3510abf795f48cc WatchSource:0}: Error finding container 22314caefce78ab1167e83b8e1851d48851d68792d9b931eb3510abf795f48cc: Status 404 returned error can't find the container with id 22314caefce78ab1167e83b8e1851d48851d68792d9b931eb3510abf795f48cc
Oct 31 16:20:37.996557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.996508    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" event={"ID":"667e8d38-e5ab-4af8-a35a-def8a1083241","Type":"ContainerStarted","Data":"a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5"}
Oct 31 16:20:37.996739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:37.996571    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" event={"ID":"667e8d38-e5ab-4af8-a35a-def8a1083241","Type":"ContainerStarted","Data":"22314caefce78ab1167e83b8e1851d48851d68792d9b931eb3510abf795f48cc"}
Oct 31 16:20:38.108996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:38.108902    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podStartSLOduration=1.108888751 podStartE2EDuration="1.108888751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:38.104900209 +0000 UTC m=+1081.804829522" watchObservedRunningTime="2026-10-31 16:20:38.108888751 +0000 UTC m=+1081.808818062"
Oct 31 16:20:38.409610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:38.409528    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:40.846483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.846429    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"]
Oct 31 16:20:40.850868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.850851    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:40.905705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.905671    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"]
Oct 31 16:20:40.945785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.945753    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbpq\" (UniqueName: \"kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:40.945968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.945941    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:40.946074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.946026    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:20:40.946074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.946042    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:40.946074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.946024    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:20:40.946213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.946077    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:20:40.947362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:40.947346    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:20:41.047490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.047437    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbpq\" (UniqueName: \"kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.047662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.047563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.047662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.047615    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.048317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.048297    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.049898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.049878    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.101235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.101174    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbpq\" (UniqueName: \"kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq\") pod \"maas-api-revoke-sub-21e0564d-cpg47\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.165241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.165209    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:41.537671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:41.537639    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"]
Oct 31 16:20:41.542229 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:20:41.542202    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd173f405_4cd6_43ac_a4b6_ed1337262ef1.slice/crio-b6e61eda548bda353d892ca469b3cddbe0b262313901bdc6136c12019e94fbd0 WatchSource:0}: Error finding container b6e61eda548bda353d892ca469b3cddbe0b262313901bdc6136c12019e94fbd0: Status 404 returned error can't find the container with id b6e61eda548bda353d892ca469b3cddbe0b262313901bdc6136c12019e94fbd0
Oct 31 16:20:42.026768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:42.026734    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47" event={"ID":"d173f405-4cd6-43ac-a4b6-ed1337262ef1","Type":"ContainerStarted","Data":"e8c78be1c5791c898c2953bbb32b3205d0bab2b6a2ac5ac5898c6de914128095"}
Oct 31 16:20:42.027231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:42.026787    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47" event={"ID":"d173f405-4cd6-43ac-a4b6-ed1337262ef1","Type":"ContainerStarted","Data":"b6e61eda548bda353d892ca469b3cddbe0b262313901bdc6136c12019e94fbd0"}
Oct 31 16:20:42.135532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:42.135411    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47" podStartSLOduration=2.135397656 podStartE2EDuration="2.135397656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:42.134560067 +0000 UTC m=+1085.834489379" watchObservedRunningTime="2026-10-31 16:20:42.135397656 +0000 UTC m=+1085.835326966"
Oct 31 16:20:43.038179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:43.038148    4519 generic.go:356] "Generic (PLEG): container finished" podID="d173f405-4cd6-43ac-a4b6-ed1337262ef1" containerID="e8c78be1c5791c898c2953bbb32b3205d0bab2b6a2ac5ac5898c6de914128095" exitCode=0
Oct 31 16:20:43.038642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:43.038241    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47" event={"ID":"d173f405-4cd6-43ac-a4b6-ed1337262ef1","Type":"ContainerDied","Data":"e8c78be1c5791c898c2953bbb32b3205d0bab2b6a2ac5ac5898c6de914128095"}
Oct 31 16:20:44.213718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.213687    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:44.285591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.285560    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca\") pod \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") "
Oct 31 16:20:44.285763 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.285642    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca\") pod \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") "
Oct 31 16:20:44.285763 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.285724    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq\" (UniqueName: \"kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq\") pod \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\" (UID: \"d173f405-4cd6-43ac-a4b6-ed1337262ef1\") "
Oct 31 16:20:44.285944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.285922    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca" pod "d173f405-4cd6-43ac-a4b6-ed1337262ef1" (UID: "d173f405-4cd6-43ac-a4b6-ed1337262ef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:20:44.286276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.286256    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:20:44.287737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.287710    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca" pod "d173f405-4cd6-43ac-a4b6-ed1337262ef1" (UID: "d173f405-4cd6-43ac-a4b6-ed1337262ef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:20:44.287821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.287771    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq" pod "d173f405-4cd6-43ac-a4b6-ed1337262ef1" (UID: "d173f405-4cd6-43ac-a4b6-ed1337262ef1"). InnerVolumeSpecName "kube-api-access-cdbpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:20:44.387332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.387244    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdbpq\" (UniqueName: \"kubernetes.io/projected/d173f405-4cd6-43ac-a4b6-ed1337262ef1-kube-api-access-cdbpq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:20:44.387332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:44.387277    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d173f405-4cd6-43ac-a4b6-ed1337262ef1-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:20:45.057650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:45.057621    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"
Oct 31 16:20:45.057826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:45.057651    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47" event={"ID":"d173f405-4cd6-43ac-a4b6-ed1337262ef1","Type":"ContainerDied","Data":"b6e61eda548bda353d892ca469b3cddbe0b262313901bdc6136c12019e94fbd0"}
Oct 31 16:20:45.057826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:45.057702    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e61eda548bda353d892ca469b3cddbe0b262313901bdc6136c12019e94fbd0"
Oct 31 16:20:47.409578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:47.409538    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:47.422072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:47.422041    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:48.088828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:48.088800    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:20:53.441837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:20:53.441802    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:20:54.005393 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:20:54.005356    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:21:04.276809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.276767    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"]
Oct 31 16:21:04.277645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.277624    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d173f405-4cd6-43ac-a4b6-ed1337262ef1" containerName="revoke-keys"
Oct 31 16:21:04.277645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.277645    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d173f405-4cd6-43ac-a4b6-ed1337262ef1" containerName="revoke-keys"
Oct 31 16:21:04.277866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.277852    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d173f405-4cd6-43ac-a4b6-ed1337262ef1" containerName="revoke-keys"
Oct 31 16:21:04.281679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.281660    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.331412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.331375    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:21:04.331412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.331394    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:21:04.331628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.331607    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:21:04.332847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.332827    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"]
Oct 31 16:21:04.375017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.374992    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:21:04.450426 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.450400    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.450620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.450443    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mmv\" (UniqueName: \"kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.450620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.450609    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.552489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.552389    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.552489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.552434    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mmv\" (UniqueName: \"kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.552688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.552526    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.553241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.553216    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.554889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.554867    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.606662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.606629    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mmv\" (UniqueName: \"kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv\") pod \"maas-api-revoke-sub-92d60d3e-75tdh\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:04.894689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:04.894601    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:05.274506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:05.274473    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"]
Oct 31 16:21:05.278385 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:05.278360    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3648f5_08e2_49f1_bb84_5c6a1d98c8fd.slice/crio-f4ed3bd42e766e070e904840a873803dfd5fc8141efc67e64fc36698d93f83ed WatchSource:0}: Error finding container f4ed3bd42e766e070e904840a873803dfd5fc8141efc67e64fc36698d93f83ed: Status 404 returned error can't find the container with id f4ed3bd42e766e070e904840a873803dfd5fc8141efc67e64fc36698d93f83ed
Oct 31 16:21:06.215734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:06.215701    4519 generic.go:356] "Generic (PLEG): container finished" podID="af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" containerID="44e9a39112259acfc3ee83ba75874fec2d0d0a1eba2b706a619b33d2863a77d4" exitCode=0
Oct 31 16:21:06.215901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:06.215758    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh" event={"ID":"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd","Type":"ContainerDied","Data":"44e9a39112259acfc3ee83ba75874fec2d0d0a1eba2b706a619b33d2863a77d4"}
Oct 31 16:21:06.215901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:06.215828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh" event={"ID":"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd","Type":"ContainerStarted","Data":"f4ed3bd42e766e070e904840a873803dfd5fc8141efc67e64fc36698d93f83ed"}
Oct 31 16:21:07.386638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.386609    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:07.389870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.389848    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca\") pod \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") "
Oct 31 16:21:07.390163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.390144    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca" pod "af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" (UID: "af3648f5-08e2-49f1-bb84-5c6a1d98c8fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:21:07.490799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.490698    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca\") pod \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") "
Oct 31 16:21:07.490799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.490793    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv\" (UniqueName: \"kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv\") pod \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\" (UID: \"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd\") "
Oct 31 16:21:07.491487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.491447    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:07.492848 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.492823    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca" pod "af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" (UID: "af3648f5-08e2-49f1-bb84-5c6a1d98c8fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:21:07.492941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.492920    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv" pod "af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" (UID: "af3648f5-08e2-49f1-bb84-5c6a1d98c8fd"). InnerVolumeSpecName "kube-api-access-w9mmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:21:07.592366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.592332    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:07.592366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:07.592364    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mmv\" (UniqueName: \"kubernetes.io/projected/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd-kube-api-access-w9mmv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:08.231831 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:08.231802    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"
Oct 31 16:21:08.232029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:08.231842    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh" event={"ID":"af3648f5-08e2-49f1-bb84-5c6a1d98c8fd","Type":"ContainerDied","Data":"f4ed3bd42e766e070e904840a873803dfd5fc8141efc67e64fc36698d93f83ed"}
Oct 31 16:21:08.232029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:08.231894    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ed3bd42e766e070e904840a873803dfd5fc8141efc67e64fc36698d93f83ed"
Oct 31 16:21:15.057627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.057590    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"]
Oct 31 16:21:15.058406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.058384    4519 cpu_manager.go:414] "containerMap: removing container" podUID="af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" containerName="revoke-keys"
Oct 31 16:21:15.058555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.058407    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" containerName="revoke-keys"
Oct 31 16:21:15.058631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.058614    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" containerName="revoke-keys"
Oct 31 16:21:15.062307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.062286    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.113969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.113934    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"]
Oct 31 16:21:15.156044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.155999    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:21:15.156239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.156163    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:21:15.156239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.156205    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:21:15.159403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.159385    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:21:15.184005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.183975    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.184223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.184204    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.184267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.184244    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmq2\" (UniqueName: \"kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.286019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.285977    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.286225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.286025    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmq2\" (UniqueName: \"kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.286225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.286079    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.286853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.286830    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.288431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.288408    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.340406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.340342    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmq2\" (UniqueName: \"kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2\") pod \"maas-api-revoke-sub-182aaf33-rb67m\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.375679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.375645    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:15.748827 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:15.748797    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"]
Oct 31 16:21:15.752563 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:15.752536    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41857a8e_f80a_43ba_92a7_0079cd8298bd.slice/crio-a0a19820818c965004b13c4c27036e95be1c3d2b9fad6e7b22b257ef5e81c18c WatchSource:0}: Error finding container a0a19820818c965004b13c4c27036e95be1c3d2b9fad6e7b22b257ef5e81c18c: Status 404 returned error can't find the container with id a0a19820818c965004b13c4c27036e95be1c3d2b9fad6e7b22b257ef5e81c18c
Oct 31 16:21:16.292196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:16.292155    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m" event={"ID":"41857a8e-f80a-43ba-92a7-0079cd8298bd","Type":"ContainerStarted","Data":"1612e7aa16a029fa7758096538718a877f9755588672a9f411eb18748b3a277e"}
Oct 31 16:21:16.292620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:16.292221    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m" event={"ID":"41857a8e-f80a-43ba-92a7-0079cd8298bd","Type":"ContainerStarted","Data":"a0a19820818c965004b13c4c27036e95be1c3d2b9fad6e7b22b257ef5e81c18c"}
Oct 31 16:21:16.403575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:16.403384    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m" podStartSLOduration=1.403371449 podStartE2EDuration="1.403371449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:16.401738048 +0000 UTC m=+1120.101667378" watchObservedRunningTime="2026-10-31 16:21:16.403371449 +0000 UTC m=+1120.103300759"
Oct 31 16:21:17.300681 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:17.300641    4519 generic.go:356] "Generic (PLEG): container finished" podID="41857a8e-f80a-43ba-92a7-0079cd8298bd" containerID="1612e7aa16a029fa7758096538718a877f9755588672a9f411eb18748b3a277e" exitCode=0
Oct 31 16:21:17.301095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:17.300694    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m" event={"ID":"41857a8e-f80a-43ba-92a7-0079cd8298bd","Type":"ContainerDied","Data":"1612e7aa16a029fa7758096538718a877f9755588672a9f411eb18748b3a277e"}
Oct 31 16:21:18.463487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.463445    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:18.528624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.528590    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca\") pod \"41857a8e-f80a-43ba-92a7-0079cd8298bd\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") "
Oct 31 16:21:18.528804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.528644    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2\" (UniqueName: \"kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2\") pod \"41857a8e-f80a-43ba-92a7-0079cd8298bd\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") "
Oct 31 16:21:18.528804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.528695    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca\") pod \"41857a8e-f80a-43ba-92a7-0079cd8298bd\" (UID: \"41857a8e-f80a-43ba-92a7-0079cd8298bd\") "
Oct 31 16:21:18.528966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.528945    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca" pod "41857a8e-f80a-43ba-92a7-0079cd8298bd" (UID: "41857a8e-f80a-43ba-92a7-0079cd8298bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:21:18.529269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.529247    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:18.530867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.530839    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca" pod "41857a8e-f80a-43ba-92a7-0079cd8298bd" (UID: "41857a8e-f80a-43ba-92a7-0079cd8298bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:21:18.530867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.530848    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2" pod "41857a8e-f80a-43ba-92a7-0079cd8298bd" (UID: "41857a8e-f80a-43ba-92a7-0079cd8298bd"). InnerVolumeSpecName "kube-api-access-dnmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:21:18.631019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.630912    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmq2\" (UniqueName: \"kubernetes.io/projected/41857a8e-f80a-43ba-92a7-0079cd8298bd-kube-api-access-dnmq2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:18.631019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:18.630971    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41857a8e-f80a-43ba-92a7-0079cd8298bd-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:19.316685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:19.316652    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"
Oct 31 16:21:19.316864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:19.316703    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m" event={"ID":"41857a8e-f80a-43ba-92a7-0079cd8298bd","Type":"ContainerDied","Data":"a0a19820818c965004b13c4c27036e95be1c3d2b9fad6e7b22b257ef5e81c18c"}
Oct 31 16:21:19.316864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:19.316773    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a19820818c965004b13c4c27036e95be1c3d2b9fad6e7b22b257ef5e81c18c"
Oct 31 16:21:20.373473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.373434    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"]
Oct 31 16:21:20.374081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.374059    4519 cpu_manager.go:414] "containerMap: removing container" podUID="41857a8e-f80a-43ba-92a7-0079cd8298bd" containerName="revoke-keys"
Oct 31 16:21:20.374081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.374077    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41857a8e-f80a-43ba-92a7-0079cd8298bd" containerName="revoke-keys"
Oct 31 16:21:20.374225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.374197    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="41857a8e-f80a-43ba-92a7-0079cd8298bd" containerName="revoke-keys"
Oct 31 16:21:20.377907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.377889    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.431312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.431282    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"]
Oct 31 16:21:20.452918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.452881    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.452928    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453080    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453248    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453308    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453337    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453373    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453420    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmdq\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.453576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.453555    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.554645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.554604    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.554823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.554801    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.554898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.554880    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.554944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.554910    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.554989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.554955    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.554989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.554980    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmdq\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555022    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555243    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555260    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555304    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555309    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555351    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555366    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.555968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.555770    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.557330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.557298    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.557432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.557305    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.609054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.609022    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.609261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.609241    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmdq\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:20.694055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:20.693959    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:21.122151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.122115    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"]
Oct 31 16:21:21.168185 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:21.168155    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059db11a_da47_4d8f_b774_236d0b42e3ed.slice/crio-8be2a370bd736f50839da746d55a7ce77524176c5af4c7be45cda0093334a684 WatchSource:0}: Error finding container 8be2a370bd736f50839da746d55a7ce77524176c5af4c7be45cda0093334a684: Status 404 returned error can't find the container with id 8be2a370bd736f50839da746d55a7ce77524176c5af4c7be45cda0093334a684
Oct 31 16:21:21.170799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.170753    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:21.170896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.170843    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:21.170896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.170871    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:21.332229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.332192    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" event={"ID":"059db11a-da47-4d8f-b774-236d0b42e3ed","Type":"ContainerStarted","Data":"2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b"}
Oct 31 16:21:21.332408 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.332252    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" event={"ID":"059db11a-da47-4d8f-b774-236d0b42e3ed","Type":"ContainerStarted","Data":"8be2a370bd736f50839da746d55a7ce77524176c5af4c7be45cda0093334a684"}
Oct 31 16:21:21.451990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.451787    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podStartSLOduration=1.451763287 podStartE2EDuration="1.451763287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:21.4452827 +0000 UTC m=+1125.145212014" watchObservedRunningTime="2026-10-31 16:21:21.451763287 +0000 UTC m=+1125.151692599"
Oct 31 16:21:21.694528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:21.694499    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:22.696033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:22.695983    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 16:21:22.696560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:22.696049    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 16:21:22.905854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:22.905818    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"]
Oct 31 16:21:22.910346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:22.910328    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:22.964358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:22.964330    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"]
Oct 31 16:21:23.005324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.005264    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"openshift-service-ca.crt\""
Oct 31 16:21:23.005324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.005312    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"default-dockercfg-8rxfz\""
Oct 31 16:21:23.006049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006027    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.006138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006126    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.006182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006158    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.006397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006372    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.006506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006486    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpq2\" (UniqueName: \"kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.006550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006529    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.006814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006799    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"distinct-2-w3-f7821a-kserve-self-signed-certs\""
Oct 31 16:21:23.006969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.006950    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"kube-root-ca.crt\""
Oct 31 16:21:23.110396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.110356    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpq2\" (UniqueName: \"kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.110689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.110658    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.110824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.110766    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.110886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.110876    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.110948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.110908    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.111098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.111082    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.111567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.111541    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.112547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.112521    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.114136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.114110    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.115903    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.118854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.118833    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.164524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.164497    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpq2\" (UniqueName: \"kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2\") pod \"distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.224208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.224113    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:23.627787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.627708    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"]
Oct 31 16:21:23.631193 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:23.631161    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16d861ab_eb14_4b79_a9a4_5b90da2f2b3a.slice/crio-392a322db379175597e61e490241f004dd398d456fa370de2b7258678e5a7387 WatchSource:0}: Error finding container 392a322db379175597e61e490241f004dd398d456fa370de2b7258678e5a7387: Status 404 returned error can't find the container with id 392a322db379175597e61e490241f004dd398d456fa370de2b7258678e5a7387
Oct 31 16:21:23.695224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.695191    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 16:21:23.695405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:23.695290    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 16:21:24.016957 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:21:24.016923    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:21:24.363709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:24.363617    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" event={"ID":"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a","Type":"ContainerStarted","Data":"a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db"}
Oct 31 16:21:24.363709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:24.363667    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" event={"ID":"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a","Type":"ContainerStarted","Data":"392a322db379175597e61e490241f004dd398d456fa370de2b7258678e5a7387"}
Oct 31 16:21:24.488479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:24.486301    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podStartSLOduration=2.486277712 podStartE2EDuration="2.486277712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:24.483703844 +0000 UTC m=+1128.183633156" watchObservedRunningTime="2026-10-31 16:21:24.486277712 +0000 UTC m=+1128.186207025"
Oct 31 16:21:24.700874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:24.700779    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 16:21:24.700874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:24.700834    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 16:21:25.225039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:25.225004    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:25.697586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:25.697109    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 16:21:25.697586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:25.697169    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 16:21:26.694715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:26.694673    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 16:21:26.695231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:26.694744    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 16:21:27.695171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:27.695135    4519 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 16:21:27.695707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:27.695197    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 16:21:28.699121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:28.699092    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:28.699561 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:28.699548    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:28.700093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:28.700071    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:21:33.224898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.224848    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:33.237741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.237712    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:33.474750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.474722    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:33.514885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.514816    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"]
Oct 31 16:21:33.519587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.519567    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.569690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.569635    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:21:33.569690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.569666    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:21:33.570056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.569737    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:21:33.570056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.569876    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:21:33.576887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.576859    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"]
Oct 31 16:21:33.598728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.598681    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.598917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.598898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.599023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.599007    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4cbt\" (UniqueName: \"kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.700773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.700734    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.700979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.700837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4cbt\" (UniqueName: \"kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.701035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.700981    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.701730 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.701705    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.703417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.703394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.754589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.754551    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4cbt\" (UniqueName: \"kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt\") pod \"maas-api-revoke-sub-476ef78f-ks9w9\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:33.833877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:33.833798    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:34.208760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:34.208640    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"]
Oct 31 16:21:34.211723 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:34.211692    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c51aa3_231c_485e_96d5_943173aecd5a.slice/crio-97f4cf2b2bc07bc6e08479db30c3b2e356c0888ed73eb7e69762dc64147955e5 WatchSource:0}: Error finding container 97f4cf2b2bc07bc6e08479db30c3b2e356c0888ed73eb7e69762dc64147955e5: Status 404 returned error can't find the container with id 97f4cf2b2bc07bc6e08479db30c3b2e356c0888ed73eb7e69762dc64147955e5
Oct 31 16:21:34.472657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:34.472624    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9" event={"ID":"91c51aa3-231c-485e-96d5-943173aecd5a","Type":"ContainerStarted","Data":"97f4cf2b2bc07bc6e08479db30c3b2e356c0888ed73eb7e69762dc64147955e5"}
Oct 31 16:21:35.481833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.481797    4519 generic.go:356] "Generic (PLEG): container finished" podID="91c51aa3-231c-485e-96d5-943173aecd5a" containerID="8f8a8c37ad4a6cdd134289def9e87f058f7dc4d3b194210d10fc14aed780f23f" exitCode=0
Oct 31 16:21:35.482369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.481907    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9" event={"ID":"91c51aa3-231c-485e-96d5-943173aecd5a","Type":"ContainerDied","Data":"8f8a8c37ad4a6cdd134289def9e87f058f7dc4d3b194210d10fc14aed780f23f"}
Oct 31 16:21:35.486773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.486748    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"]
Oct 31 16:21:35.504164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.504133    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.534290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.534263    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"]
Oct 31 16:21:35.539419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.539399    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.585271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.585243    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"]
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.628841    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.628938    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.628973    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629034    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629083    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629129    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629216    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629267    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629295    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629371    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.629506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.629433    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkmj\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.630409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.630379    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lznj\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.630595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.630446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.630595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.630506    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.630725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.630590    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.630915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.630885    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.632234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.632211    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"]
Oct 31 16:21:35.732920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.732829    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.732920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.732896    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.732942    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.732979    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733023    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733062    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733106    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733139    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733182    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733216    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkmj\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733280    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733314    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lznj\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733346    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733362    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733400    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733442    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733510    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.733611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.734112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733614    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.734112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733633    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.734112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733671    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.734112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.733944    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.734112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.734009    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.734112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.734013    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.734421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.734306    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.734533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.734511    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.734645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.734620    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.735656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.735629    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.736148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.736126    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.736508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.736485    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.736657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.736636    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.787105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.787065    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.787278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.787151    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkmj\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:35.787278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.787249    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.787411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.787366    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lznj\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.818345 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.818317    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:35.853948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:35.853920    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:36.050674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.050599    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"]
Oct 31 16:21:36.054608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.054585    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2912257c-txmb2"]
Oct 31 16:21:36.257201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.257169    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"]
Oct 31 16:21:36.263248 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:36.263214    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ca8c18_1992_4714_93c3_6861075fdb02.slice/crio-061f52ccfcee28e77360ce782669492f3087f74c047a9b795884b93f697e46d7 WatchSource:0}: Error finding container 061f52ccfcee28e77360ce782669492f3087f74c047a9b795884b93f697e46d7: Status 404 returned error can't find the container with id 061f52ccfcee28e77360ce782669492f3087f74c047a9b795884b93f697e46d7
Oct 31 16:21:36.264785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.264761    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"]
Oct 31 16:21:36.266290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.266253    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:36.266356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.266318    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:36.266392 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.266351    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:36.266510 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:36.266490    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbe4ce9_491e_41ea_bb87_0f09d5ec374a.slice/crio-d32706adbb7012ffbbe06f9c59e7eb4a0498f7fa4a2554d022d29afc70072aaf WatchSource:0}: Error finding container d32706adbb7012ffbbe06f9c59e7eb4a0498f7fa4a2554d022d29afc70072aaf: Status 404 returned error can't find the container with id d32706adbb7012ffbbe06f9c59e7eb4a0498f7fa4a2554d022d29afc70072aaf
Oct 31 16:21:36.268747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.268719    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:36.268847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.268784    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:36.268847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.268826    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:36.445627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.445592    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3502a88c-3568-4a72-8e5a-36bdd99914d6" path="/var/lib/kubelet/pods/3502a88c-3568-4a72-8e5a-36bdd99914d6/volumes"
Oct 31 16:21:36.492635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.492592    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" event={"ID":"01ca8c18-1992-4714-93c3-6861075fdb02","Type":"ContainerStarted","Data":"e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0"}
Oct 31 16:21:36.493127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.492655    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" event={"ID":"01ca8c18-1992-4714-93c3-6861075fdb02","Type":"ContainerStarted","Data":"061f52ccfcee28e77360ce782669492f3087f74c047a9b795884b93f697e46d7"}
Oct 31 16:21:36.495209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.495133    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" event={"ID":"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a","Type":"ContainerStarted","Data":"2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc"}
Oct 31 16:21:36.495209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.495180    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" event={"ID":"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a","Type":"ContainerStarted","Data":"d32706adbb7012ffbbe06f9c59e7eb4a0498f7fa4a2554d022d29afc70072aaf"}
Oct 31 16:21:36.671313 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.671287    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:36.720824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.720775    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podStartSLOduration=1.7207593509999999 podStartE2EDuration="1.720759351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:36.609871092 +0000 UTC m=+1140.309800407" watchObservedRunningTime="2026-10-31 16:21:36.720759351 +0000 UTC m=+1140.420688664"
Oct 31 16:21:36.725102 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.725063    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podStartSLOduration=1.725050789 podStartE2EDuration="1.725050789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:36.719881592 +0000 UTC m=+1140.419810904" watchObservedRunningTime="2026-10-31 16:21:36.725050789 +0000 UTC m=+1140.424980167"
Oct 31 16:21:36.747678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.747651    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca\") pod \"91c51aa3-231c-485e-96d5-943173aecd5a\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") "
Oct 31 16:21:36.747859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.747842    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca\") pod \"91c51aa3-231c-485e-96d5-943173aecd5a\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") "
Oct 31 16:21:36.747959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.747935    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt\" (UniqueName: \"kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt\") pod \"91c51aa3-231c-485e-96d5-943173aecd5a\" (UID: \"91c51aa3-231c-485e-96d5-943173aecd5a\") "
Oct 31 16:21:36.748241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.748210    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca" pod "91c51aa3-231c-485e-96d5-943173aecd5a" (UID: "91c51aa3-231c-485e-96d5-943173aecd5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:21:36.748723 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.748698    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:36.750071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.750047    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt" pod "91c51aa3-231c-485e-96d5-943173aecd5a" (UID: "91c51aa3-231c-485e-96d5-943173aecd5a"). InnerVolumeSpecName "kube-api-access-k4cbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:21:36.750307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.750274    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca" pod "91c51aa3-231c-485e-96d5-943173aecd5a" (UID: "91c51aa3-231c-485e-96d5-943173aecd5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:21:36.818748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.818678    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:36.849881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.849851    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4cbt\" (UniqueName: \"kubernetes.io/projected/91c51aa3-231c-485e-96d5-943173aecd5a-kube-api-access-k4cbt\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:36.849881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.849878    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c51aa3-231c-485e-96d5-943173aecd5a-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:36.854272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:36.854240    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:37.523185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.517439    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"
Oct 31 16:21:37.523709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.523565    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9" event={"ID":"91c51aa3-231c-485e-96d5-943173aecd5a","Type":"ContainerDied","Data":"97f4cf2b2bc07bc6e08479db30c3b2e356c0888ed73eb7e69762dc64147955e5"}
Oct 31 16:21:37.523709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.523629    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97f4cf2b2bc07bc6e08479db30c3b2e356c0888ed73eb7e69762dc64147955e5"
Oct 31 16:21:37.820020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.819912    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:37.820020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.819991    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:37.855954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.855911    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:37.856111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:37.855969    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:38.818968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:38.818929    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:38.819599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:38.819561    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:38.855044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:38.855011    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:38.855167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:38.855068    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:39.726975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:39.726942    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:21:39.818926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:39.818889    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:39.819188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:39.819155    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:39.854649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:39.854618    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:39.854794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:39.854684    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:40.820480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:40.819851    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:40.820480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:40.819914    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:40.860483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:40.855837    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:40.860483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:40.855911    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:41.427496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.412561    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.427496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.415553    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.465481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.464580    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:21:41.512190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.511344    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.572487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.571556    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.620251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.620207    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.679637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.679536    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.692593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.692541    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.756773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.755568    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:41.788601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.788556    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:21:41.848773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.822294    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:41.848773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.822397    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:41.893480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.873272    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:41.894341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.894293    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:41.972552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:41.947548    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.097480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.097220    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.130472 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.129570    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.384539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.383580    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:42.384539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.383649    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.419482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.417979    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.419482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.418437    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:21:42.424479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.420678    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:21:42.424479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.421026    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:21:42.469478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.468730    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.469478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.469136    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:42.517490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.516535    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.517490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.516981    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:21:42.578805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.577554    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:21:42.578805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.577962    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:21:42.754517 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.753734    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:42.754517 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.753796    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.761515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.756662    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:21:42.792514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.761822    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:21:42.792514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.763006    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.792514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.773681    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.792514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.775892    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:21:42.792514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.775953    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:21:42.852506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.837651    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:21:42.852506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.838940    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:21:42.852506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.840268    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:42.852506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.840305    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:42.886479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.872920    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:42.886479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.872976    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:42.971480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.930706    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.001507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:42.956896    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.002187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.002156    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:21:43.136706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.108803    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.136706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.109224    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:21:43.165493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.164587    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.165493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.165006    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:21:43.480414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.480230    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.521506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.496478    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:21:43.521506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.498615    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.522113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.522068    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:21:43.600477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.600064    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:43.600477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.600154    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.662237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.610971    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.662237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.633756    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 31 16:21:43.772530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.759526    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.772530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.766947    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.786478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.777518    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.863793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.863737    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:21:43.864386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.864353    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:43.864506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.864421    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:43.911027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.882934    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:43.911027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:43.882994    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:44.037533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.024571    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.149107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.122561    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:21:44.149107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.122630    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.218515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.182767    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.242273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.241728    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:21:44.253496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.252577    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:21:44.256482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.253717    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:21:44.256482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.254795    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:21:44.256482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.254838    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:21:44.259001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.258981    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:21:44.267927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.267905    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:21:44.277602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.277580    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:21:44.310078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.309984    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:21:44.339924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.337714    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:21:44.358380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.349652    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:21:44.819062 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.819015    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 16:21:44.819259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.819084    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 16:21:44.854727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.854685    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 16:21:44.854895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:44.854752    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 16:21:46.100249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:46.100209    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:46.101184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:46.101161    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:46.101799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:46.101784    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:21:46.897334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:46.897269    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:46.897577 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:46.897365    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:47.108132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:47.108071    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:47.109450 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:47.109425    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:47.139924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:47.139896    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:21:49.194670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.194632    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"]
Oct 31 16:21:49.195611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.195593    4519 cpu_manager.go:414] "containerMap: removing container" podUID="91c51aa3-231c-485e-96d5-943173aecd5a" containerName="revoke-keys"
Oct 31 16:21:49.195659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.195612    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c51aa3-231c-485e-96d5-943173aecd5a" containerName="revoke-keys"
Oct 31 16:21:49.195773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.195762    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c51aa3-231c-485e-96d5-943173aecd5a" containerName="revoke-keys"
Oct 31 16:21:49.201341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.201323    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.250948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.250909    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"]
Oct 31 16:21:49.251383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.251351    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:21:49.290920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.290891    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:21:49.290920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.290903    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:21:49.291131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.290974    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:21:49.323980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.323949    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.324127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.324078    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg25p\" (UniqueName: \"kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.324195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.324187    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.425831 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.425793    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.426036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.425912    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg25p\" (UniqueName: \"kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.426036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.426021    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.426671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.426650    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.428103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.428084    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.480494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.480433    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg25p\" (UniqueName: \"kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p\") pod \"maas-api-revoke-sub-64b3ee55-h5wn2\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.515113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.515085    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:49.891993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:49.891916    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"]
Oct 31 16:21:49.896150 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:49.896111    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8eaac28_479b_4cde_8c80_3977515ca30a.slice/crio-b4fe4a68de22008c984b991c93dbbe926ad4c486cf52345924df1846268e52eb WatchSource:0}: Error finding container b4fe4a68de22008c984b991c93dbbe926ad4c486cf52345924df1846268e52eb: Status 404 returned error can't find the container with id b4fe4a68de22008c984b991c93dbbe926ad4c486cf52345924df1846268e52eb
Oct 31 16:21:50.081049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:50.081019    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2" event={"ID":"c8eaac28-479b-4cde-8c80-3977515ca30a","Type":"ContainerStarted","Data":"b4fe4a68de22008c984b991c93dbbe926ad4c486cf52345924df1846268e52eb"}
Oct 31 16:21:51.089440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:51.089345    4519 generic.go:356] "Generic (PLEG): container finished" podID="c8eaac28-479b-4cde-8c80-3977515ca30a" containerID="dc30cb1d73bda110fb9fa9f3f117b2069098c88199891d6f73cc7cdfc0e61605" exitCode=0
Oct 31 16:21:51.089440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:51.089382    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2" event={"ID":"c8eaac28-479b-4cde-8c80-3977515ca30a","Type":"ContainerDied","Data":"dc30cb1d73bda110fb9fa9f3f117b2069098c88199891d6f73cc7cdfc0e61605"}
Oct 31 16:21:52.046018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.045977    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"]
Oct 31 16:21:52.092587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.092558    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41a120d8-r6rqt"]
Oct 31 16:21:52.281232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.281205    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:52.368665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.368571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca\") pod \"c8eaac28-479b-4cde-8c80-3977515ca30a\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") "
Oct 31 16:21:52.368840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.368745    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p\" (UniqueName: \"kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p\") pod \"c8eaac28-479b-4cde-8c80-3977515ca30a\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") "
Oct 31 16:21:52.368906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.368860    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca\") pod \"c8eaac28-479b-4cde-8c80-3977515ca30a\" (UID: \"c8eaac28-479b-4cde-8c80-3977515ca30a\") "
Oct 31 16:21:52.369209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.369186    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca" pod "c8eaac28-479b-4cde-8c80-3977515ca30a" (UID: "c8eaac28-479b-4cde-8c80-3977515ca30a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:21:52.369428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.369410    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:52.370669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.370645    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca" pod "c8eaac28-479b-4cde-8c80-3977515ca30a" (UID: "c8eaac28-479b-4cde-8c80-3977515ca30a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:21:52.370772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.370753    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p" pod "c8eaac28-479b-4cde-8c80-3977515ca30a" (UID: "c8eaac28-479b-4cde-8c80-3977515ca30a"). InnerVolumeSpecName "kube-api-access-jg25p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:21:52.440677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.440644    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a" path="/var/lib/kubelet/pods/cc9e1cf1-a092-4c34-b0d8-5994e9db4f8a/volumes"
Oct 31 16:21:52.471196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.471171    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8eaac28-479b-4cde-8c80-3977515ca30a-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:52.471196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:52.471197    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg25p\" (UniqueName: \"kubernetes.io/projected/c8eaac28-479b-4cde-8c80-3977515ca30a-kube-api-access-jg25p\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:21:53.109334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:53.109284    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2" event={"ID":"c8eaac28-479b-4cde-8c80-3977515ca30a","Type":"ContainerDied","Data":"b4fe4a68de22008c984b991c93dbbe926ad4c486cf52345924df1846268e52eb"}
Oct 31 16:21:53.109953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:53.109932    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4fe4a68de22008c984b991c93dbbe926ad4c486cf52345924df1846268e52eb"
Oct 31 16:21:53.110129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:53.110116    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"
Oct 31 16:21:54.004593 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:21:54.004558    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:21:58.045030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.044952    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"]
Oct 31 16:21:58.092056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.092018    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0999d3-dvpp5"]
Oct 31 16:21:58.440492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.440390    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ffee20-bf1e-4fbf-9bce-2d74736d8544" path="/var/lib/kubelet/pods/e6ffee20-bf1e-4fbf-9bce-2d74736d8544/volumes"
Oct 31 16:21:58.780643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.780603    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"]
Oct 31 16:21:58.781242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.781229    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c8eaac28-479b-4cde-8c80-3977515ca30a" containerName="revoke-keys"
Oct 31 16:21:58.781281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.781243    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8eaac28-479b-4cde-8c80-3977515ca30a" containerName="revoke-keys"
Oct 31 16:21:58.781373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.781363    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8eaac28-479b-4cde-8c80-3977515ca30a" containerName="revoke-keys"
Oct 31 16:21:58.785310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.785291    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.837386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.837351    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"]
Oct 31 16:21:58.856774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.856739    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.856947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.856860    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.856947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.856932    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65kz\" (UniqueName: \"kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.879765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.879725    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:21:58.879901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.879772    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:21:58.886042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.886012    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:21:58.886042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.886039    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:21:58.957898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.957871    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.958061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.957919    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.958061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.957955    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65kz\" (UniqueName: \"kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.958570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.958548    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:58.960516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:58.960488    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:59.011953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:59.011927    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65kz\" (UniqueName: \"kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz\") pod \"maas-api-revoke-sub-a06b5dca-ptmd8\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:59.099254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:59.099179    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:21:59.472113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:21:59.472083    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"]
Oct 31 16:21:59.476553 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:21:59.476527    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0162124_ae0f_4f68_9410_a67166726c76.slice/crio-711ca408c49cf7596dbb42a064e1fefdca9c8f146ffcb3074aa69fb988efbc81 WatchSource:0}: Error finding container 711ca408c49cf7596dbb42a064e1fefdca9c8f146ffcb3074aa69fb988efbc81: Status 404 returned error can't find the container with id 711ca408c49cf7596dbb42a064e1fefdca9c8f146ffcb3074aa69fb988efbc81
Oct 31 16:22:00.174441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:00.174361    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8" event={"ID":"b0162124-ae0f-4f68-9410-a67166726c76","Type":"ContainerStarted","Data":"e2b7bbb8d62050dfebaf96901faac7e489e54d36cde07e410161d16b6f5378ad"}
Oct 31 16:22:00.174836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:00.174479    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8" event={"ID":"b0162124-ae0f-4f68-9410-a67166726c76","Type":"ContainerStarted","Data":"711ca408c49cf7596dbb42a064e1fefdca9c8f146ffcb3074aa69fb988efbc81"}
Oct 31 16:22:00.286275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:00.286224    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8" podStartSLOduration=2.286210097 podStartE2EDuration="2.286210097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:00.281730452 +0000 UTC m=+1163.981659762" watchObservedRunningTime="2026-10-31 16:22:00.286210097 +0000 UTC m=+1163.986139407"
Oct 31 16:22:01.182918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:01.182878    4519 generic.go:356] "Generic (PLEG): container finished" podID="b0162124-ae0f-4f68-9410-a67166726c76" containerID="e2b7bbb8d62050dfebaf96901faac7e489e54d36cde07e410161d16b6f5378ad" exitCode=0
Oct 31 16:22:01.183321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:01.182919    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8" event={"ID":"b0162124-ae0f-4f68-9410-a67166726c76","Type":"ContainerDied","Data":"e2b7bbb8d62050dfebaf96901faac7e489e54d36cde07e410161d16b6f5378ad"}
Oct 31 16:22:02.355144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.355120    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:22:02.509513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.509477    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca\") pod \"b0162124-ae0f-4f68-9410-a67166726c76\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") "
Oct 31 16:22:02.509705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.509535    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz\" (UniqueName: \"kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz\") pod \"b0162124-ae0f-4f68-9410-a67166726c76\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") "
Oct 31 16:22:02.509705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.509659    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca\") pod \"b0162124-ae0f-4f68-9410-a67166726c76\" (UID: \"b0162124-ae0f-4f68-9410-a67166726c76\") "
Oct 31 16:22:02.510075 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.510043    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca" pod "b0162124-ae0f-4f68-9410-a67166726c76" (UID: "b0162124-ae0f-4f68-9410-a67166726c76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:02.511687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.511662    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca" pod "b0162124-ae0f-4f68-9410-a67166726c76" (UID: "b0162124-ae0f-4f68-9410-a67166726c76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:02.511976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.511955    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz" pod "b0162124-ae0f-4f68-9410-a67166726c76" (UID: "b0162124-ae0f-4f68-9410-a67166726c76"). InnerVolumeSpecName "kube-api-access-h65kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:02.611381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.611343    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:02.611381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.611373    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0162124-ae0f-4f68-9410-a67166726c76-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:02.611381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:02.611389    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h65kz\" (UniqueName: \"kubernetes.io/projected/b0162124-ae0f-4f68-9410-a67166726c76-kube-api-access-h65kz\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:03.200196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:03.200157    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8" event={"ID":"b0162124-ae0f-4f68-9410-a67166726c76","Type":"ContainerDied","Data":"711ca408c49cf7596dbb42a064e1fefdca9c8f146ffcb3074aa69fb988efbc81"}
Oct 31 16:22:03.200388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:03.200214    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711ca408c49cf7596dbb42a064e1fefdca9c8f146ffcb3074aa69fb988efbc81"
Oct 31 16:22:03.200388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:03.200180    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"
Oct 31 16:22:04.513262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.513231    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"]
Oct 31 16:22:04.513897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.513883    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b0162124-ae0f-4f68-9410-a67166726c76" containerName="revoke-keys"
Oct 31 16:22:04.513944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.513898    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0162124-ae0f-4f68-9410-a67166726c76" containerName="revoke-keys"
Oct 31 16:22:04.514017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.514007    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0162124-ae0f-4f68-9410-a67166726c76" containerName="revoke-keys"
Oct 31 16:22:04.517915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.517900    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541375    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541444    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541588    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541692    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541767    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541843    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.541900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541871    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.542018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.541990    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7d2\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.542117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.542094    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.630895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.630858    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"]
Oct 31 16:22:04.643185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643148    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643188    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643218    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643243    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643307    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7d2\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643380    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643440    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643526    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.643662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.643576    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.644194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.644067    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.644194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.644014    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.644917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.644867    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.645236 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.645216    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.645638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.645590    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.646357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.646334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.647968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.647943    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.697172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.697143    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.697598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.697576    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7d2\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.832282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.832186    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:04.926880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.926790    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"]
Oct 31 16:22:04.934766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.934740    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.948562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm44t\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.948629    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.948884    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.948928    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.948978    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.949021    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.949071    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.949153    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.949742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.949225    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:04.954673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.954585    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"]
Oct 31 16:22:04.960065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.960043    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:04.989241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:04.989213    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"]
Oct 31 16:22:05.013145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.013100    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"]
Oct 31 16:22:05.050220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050184    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050228    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.050413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050329    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050380    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050435    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm44t\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.050623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050492    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.050623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050518    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050563    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050595    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050673    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.050878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050741    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.050878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050801    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.050878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050859    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h75mn\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.051039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050881    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.050936    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051002    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051062    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051137    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.051250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051189    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051266    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051326    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051355    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.051684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.051662    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.052937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.052911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.053079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.053062    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.104192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.104109    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.104192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.104162    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm44t\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.152550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.152511    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.152710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.152622    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.152762 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.152709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.152762 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.152752    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.152887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.152785    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153133    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153161    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153206    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153211    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153319    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153428    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75mn\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153633    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.153775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.153736    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.155489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.155436    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.155641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.155624    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.206849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.206820    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.207184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.207162    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75mn\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.258659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.258627    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:05.276043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.275985    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"]
Oct 31 16:22:05.278173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.278143    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:05.279247 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:05.279219    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa01192e_c28e_43c6_8780_7afcb8be04a7.slice/crio-e079c4994759f38b991aec7cee5003996ef9562cb4e8db068c7a3dd93d8b01bf WatchSource:0}: Error finding container e079c4994759f38b991aec7cee5003996ef9562cb4e8db068c7a3dd93d8b01bf: Status 404 returned error can't find the container with id e079c4994759f38b991aec7cee5003996ef9562cb4e8db068c7a3dd93d8b01bf
Oct 31 16:22:05.282722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.282688    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.282819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.282753    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.282819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.282784    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.645751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.645659    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"]
Oct 31 16:22:05.649887 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:05.649853    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014bdfbf_4d83_45f1_8b5f_302c38a61565.slice/crio-ae939ebbc5251a8cd86bb969ea3af2c4071f94c19cf834e390f159dbf86a75c5 WatchSource:0}: Error finding container ae939ebbc5251a8cd86bb969ea3af2c4071f94c19cf834e390f159dbf86a75c5: Status 404 returned error can't find the container with id ae939ebbc5251a8cd86bb969ea3af2c4071f94c19cf834e390f159dbf86a75c5
Oct 31 16:22:05.652659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.652618    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.652749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.652709    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.652799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.652757    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.697506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.697482    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"]
Oct 31 16:22:05.706021 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:05.705967    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f04b95_169f_4b49_8af6_f48c677b6def.slice/crio-604a670f1ae9842e5531951e3d3331e5ae956b59636e5a189da67d3139190ae1 WatchSource:0}: Error finding container 604a670f1ae9842e5531951e3d3331e5ae956b59636e5a189da67d3139190ae1: Status 404 returned error can't find the container with id 604a670f1ae9842e5531951e3d3331e5ae956b59636e5a189da67d3139190ae1
Oct 31 16:22:05.709714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.709429    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.709714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.709544    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.709714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.709580    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.727498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.727445    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"]
Oct 31 16:22:05.733367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.733343    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.765686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.765653    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.765792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.765705    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2cq\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.765840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.765825    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.765894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.765876    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.766073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.766049    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.766128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.766105    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.766244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.766225    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.766323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.766306    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.766381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.766353    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.803176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.803139    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"]
Oct 31 16:22:05.867503 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867421    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867503 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867484    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2cq\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867527    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867571    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867654    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867692    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867764    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867828    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.867881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.867872    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.868757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.868732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.869390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.869367    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.869619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.869601    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.869815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.869798    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.872491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.871790    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.872491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.872029    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.872491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.872086    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.922434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.922361    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:05.922790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:05.922768    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2cq\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:06.132479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.130756    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:06.248829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.248796    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" event={"ID":"014bdfbf-4d83-45f1-8b5f-302c38a61565","Type":"ContainerStarted","Data":"d1f7bea4293db9d76ebc2ea4395e487be39a9c08c87f825ca0012db955b2b713"}
Oct 31 16:22:06.249029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.248880    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" event={"ID":"014bdfbf-4d83-45f1-8b5f-302c38a61565","Type":"ContainerStarted","Data":"ae939ebbc5251a8cd86bb969ea3af2c4071f94c19cf834e390f159dbf86a75c5"}
Oct 31 16:22:06.260258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.259865    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:06.261273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.261059    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:06.261273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.261108    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:06.305813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.305624    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" event={"ID":"aa01192e-c28e-43c6-8780-7afcb8be04a7","Type":"ContainerStarted","Data":"82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859"}
Oct 31 16:22:06.305813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.305706    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" event={"ID":"aa01192e-c28e-43c6-8780-7afcb8be04a7","Type":"ContainerStarted","Data":"e079c4994759f38b991aec7cee5003996ef9562cb4e8db068c7a3dd93d8b01bf"}
Oct 31 16:22:06.309783 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.309682    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" event={"ID":"d8f04b95-169f-4b49-8af6-f48c677b6def","Type":"ContainerStarted","Data":"91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462"}
Oct 31 16:22:06.309926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.309747    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" event={"ID":"d8f04b95-169f-4b49-8af6-f48c677b6def","Type":"ContainerStarted","Data":"604a670f1ae9842e5531951e3d3331e5ae956b59636e5a189da67d3139190ae1"}
Oct 31 16:22:06.365023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.364974    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podStartSLOduration=2.364959706 podStartE2EDuration="2.364959706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:06.361691621 +0000 UTC m=+1170.061620932" watchObservedRunningTime="2026-10-31 16:22:06.364959706 +0000 UTC m=+1170.064889018"
Oct 31 16:22:06.493387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.493254    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podStartSLOduration=2.493233579 podStartE2EDuration="2.493233579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:06.492320302 +0000 UTC m=+1170.192249614" watchObservedRunningTime="2026-10-31 16:22:06.493233579 +0000 UTC m=+1170.193162891"
Oct 31 16:22:06.547427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.547396    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"]
Oct 31 16:22:06.552173 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:06.552139    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba0ba18_0be8_4c12_939d_0f3800da54bf.slice/crio-f0aa494f323d2e39fb4ed32f5a1ab0de12a37476bca74a11ef8bb0ffc67e579b WatchSource:0}: Error finding container f0aa494f323d2e39fb4ed32f5a1ab0de12a37476bca74a11ef8bb0ffc67e579b: Status 404 returned error can't find the container with id f0aa494f323d2e39fb4ed32f5a1ab0de12a37476bca74a11ef8bb0ffc67e579b
Oct 31 16:22:06.555017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.554980    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:06.555118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.555054    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:06.555118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.555094    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:06.608753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.608703    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podStartSLOduration=2.608686339 podStartE2EDuration="2.608686339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:06.604382957 +0000 UTC m=+1170.304312269" watchObservedRunningTime="2026-10-31 16:22:06.608686339 +0000 UTC m=+1170.308615650"
Oct 31 16:22:06.838540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.837513    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:06.853492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.852024    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:06.853492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:06.852086    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:07.263478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.259850    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:07.263478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.259908    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:07.280479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.279511    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:07.284478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.283550    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:07.284478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.283610    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:07.440546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.438795    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" event={"ID":"eba0ba18-0be8-4c12-939d-0f3800da54bf","Type":"ContainerStarted","Data":"e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941"}
Oct 31 16:22:07.440546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.438897    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" event={"ID":"eba0ba18-0be8-4c12-939d-0f3800da54bf","Type":"ContainerStarted","Data":"f0aa494f323d2e39fb4ed32f5a1ab0de12a37476bca74a11ef8bb0ffc67e579b"}
Oct 31 16:22:07.551957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.551844    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podStartSLOduration=2.551829153 podStartE2EDuration="2.551829153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:07.550645846 +0000 UTC m=+1171.250575156" watchObservedRunningTime="2026-10-31 16:22:07.551829153 +0000 UTC m=+1171.251758464"
Oct 31 16:22:07.833115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.833016    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:07.833402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:07.833359    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:08.065546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.065513    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"]
Oct 31 16:22:08.093227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.093137    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76bd0183-bmdlt"]
Oct 31 16:22:08.144924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.144891    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:08.160147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.160098    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.160444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.160403    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:08.270481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.267530    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.270481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.267588    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:08.294490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.284817    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.294490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.284873    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:08.499479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.485557    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5189a7-0635-46b2-9b7e-f5601e684119" path="/var/lib/kubelet/pods/8f5189a7-0635-46b2-9b7e-f5601e684119/volumes"
Oct 31 16:22:08.796662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.796556    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded"
Oct 31 16:22:08.824664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.824621    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:08.834863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.834827    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.835126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.835098    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:08.865598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.865550    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:08.964483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:08.963841    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.075673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.075561    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.132933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.132886    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:22:09.134537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.134509    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.158702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.158632    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:09.187480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.186672    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.270479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.261789    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.270479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.261860    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:09.270479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.267108    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.270479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.267431    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:22:09.270479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.269113    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.290479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.288838    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.290479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.288908    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:09.326155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.326053    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:22:09.506478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.505746    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.533481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.532544    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.784622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.784574    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Oct 31 16:22:09.828222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.828172    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:09.828905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.828880    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:22:09.836592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.836562    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.836797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.836772    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:09.909479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.909363    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:22:09.913477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:09.909770    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:22:10.142542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.111582    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.193497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.140076    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:10.193497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.149909    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:22:10.193497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.149978    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:22:10.193497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.148604    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.193497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.160574    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.276543    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.276947    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.277202    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.286715    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.311636    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.311708    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:22:10.340804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.311765    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:22:10.368540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.344193    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.368540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.350247    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:22:10.418605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.373027    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.418605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.410646    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.418605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.410701    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:10.418605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.410796    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:22:10.466306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.466177    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.466306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.466235    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:10.519507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.506548    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:22:10.536276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.532711    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:10.819475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.819231    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:22:10.819475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.819294    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"]
Oct 31 16:22:10.836998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.835756    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:10.892854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.892246    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.892854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.892302    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:10.903187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.903157    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:22:10.904280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.904258    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:10.940478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.909513    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:10.986827 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.986781    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:22:11.030480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:10.987106    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:22:11.119560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.065924    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:22:11.119560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.066015    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:22:11.119560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.066049    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"]
Oct 31 16:22:11.136689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.136645    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.138620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.138591    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.139220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.139198    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrjl\" (UniqueName: \"kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.164480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.161632    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:11.164480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.162573    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.164480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.162620    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:11.166654 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.166624    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:22:11.279736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.276823    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.280434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.277113    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.280802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.277862    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.281499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.278363    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.281639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.281617    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:11.297893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.278637    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.301534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.301501    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:11.301694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.296082    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.301837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.301821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrjl\" (UniqueName: \"kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.330381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.330346    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.330729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.302505    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd3ffc3-r4m9v"]
Oct 31 16:22:11.330860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.294988    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.332734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.332711    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.333280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.333260    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:22:11.364479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.347746    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.381794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.381693    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.395487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.390966    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.418771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.418723    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"]
Oct 31 16:22:11.463292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.441301    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrjl\" (UniqueName: \"kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl\") pod \"maas-api-revoke-sub-56adc755-rmvht\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.557486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.515482    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.582479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.558906    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:22:11.597479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.573412    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:11.628690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.628648    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 16:22:11.834479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.833472    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.834479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.833533    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:11.867510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.867470    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.910326 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:11.910248    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:12.147122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.147088    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.147432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.147386    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:12.182674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.182629    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.285477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.260518    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.285477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.260578    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:12.288371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.286153    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.318487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.286298    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.318487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.286326    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.318487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.286360    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.384423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.384381    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.386932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.385326    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:12.386932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.385637    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.468567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.440398    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.468567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.440493    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:12.561846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.561788    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.591668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.591621    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:22:12.665506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.664427    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d769e43-791b-4d5c-b7ef-c1b168246c71" path="/var/lib/kubelet/pods/9d769e43-791b-4d5c-b7ef-c1b168246c71/volumes"
Oct 31 16:22:12.837882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.837782    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.838162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.838129    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:12.992479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.992222    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:12.992692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:12.992658    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:13.165820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.165667    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:13.243481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.213098    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.252473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.248679    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:13.252473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.245755    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:13.252473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.229924    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:22:13.252830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.252809    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:22:13.455826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.455539    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:22:13.455826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.455642    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:22:13.456948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.456916    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:13.459161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.459136    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:22:13.584249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.568226    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:22:13.630896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.630681    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:13.714563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.689022    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.714563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.689083    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:13.726632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.726591    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.726872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.726844    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:13.758484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.757634    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:22:13.760796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.760768    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:22:13.813299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.811771    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:22:13.813299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.812642    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:22:13.856543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.856502    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:13.888491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.865445    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.888491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.865515    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 16:22:13.903491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.894204    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:22:13.903491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.894263    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:22:13.933152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.933122    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:22:13.940294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.940058    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:22:13.941346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.941327    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:22:13.962267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:13.962240    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:22:14.136421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.136321    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"]
Oct 31 16:22:14.138628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.137162    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:14.138628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.137212    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:14.139574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.139555    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"]
Oct 31 16:22:14.149853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.149833    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"]
Oct 31 16:22:14.150034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.149954    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.255764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.253883    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.255764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.253951    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbg5\" (UniqueName: \"kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.255764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.254295    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.259013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.258985    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 16:22:14.261023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.260992    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 16:22:14.308536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.283295    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 16:22:14.308801 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.308765    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 16:22:14.359766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.357698    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.359766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.357768    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbg5\" (UniqueName: \"kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.359766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.358063    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.359766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.358974    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.495622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.495593    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbg5\" (UniqueName: \"kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:14.591543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:14.573292    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bf0f66-ctbjv\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:15.078479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.035712    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:15.214479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.212271    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:15.214479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.212331    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:15.383537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.382664    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:15.409119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.409089    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:15.488337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.488295    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:22:15.633479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.610548    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:15.633479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.631964    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:15.656564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.600768    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:22:15.730679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.730635    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht" event={"ID":"87a645cb-02bf-47c5-8b80-b4882c50d9a2","Type":"ContainerStarted","Data":"05493aafdcaa13593fab5132e5eccecff7a55efb646656ecf2cd64206020c989"}
Oct 31 16:22:15.996427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:15.996385    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.148480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.113806    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:22:16.158378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.157868    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:22:16.158378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.157950    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 31 16:22:16.257564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.251858    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:16.292725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.287770    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:16.292725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.287851    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.292725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.288245    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:16.292725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.288297    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.292725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.288856    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.292725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.289137    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:22:16.320498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.320171    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.346188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.346142    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.412946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.412911    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:16.450426 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.414518    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.451081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.414589    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:22:16.872871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.856718    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.872871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.859128    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:16.889131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:16.889084    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.023081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.020880    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:22:17.196480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.191868    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:17.196480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.191928    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.221493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.196811    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.428747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.425812    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.447829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.447785    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.454521 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:17.454493    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.008s"
Oct 31 16:22:17.454803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.454785    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:17.454958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.454944    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:17.455114 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455101    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:22:17.455269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455256    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:22:17.455410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455398    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:22:17.455552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455539    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:22:17.455675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455664    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:22:17.455803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455792    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:22:17.455936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.455925    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:22:17.461550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.461526    4519 generic.go:356] "Generic (PLEG): container finished" podID="87a645cb-02bf-47c5-8b80-b4882c50d9a2" containerID="62f244fd93f882f75568090f7dfa71581bac22da4419474f8d220dc3226fb1ff" exitCode=0
Oct 31 16:22:17.463373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.463359    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:17.543676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.543525    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht" event={"ID":"87a645cb-02bf-47c5-8b80-b4882c50d9a2","Type":"ContainerDied","Data":"62f244fd93f882f75568090f7dfa71581bac22da4419474f8d220dc3226fb1ff"}
Oct 31 16:22:17.543676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.543612    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:22:17.609575 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:17.609530    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f6b22a_0d41_4281_a055_d6b1ab7ff2b0.slice/crio-eb144c4129e29b4190a3aa909332f994a85f844d211a9709f7542c3f43cf3e7f WatchSource:0}: Error finding container eb144c4129e29b4190a3aa909332f994a85f844d211a9709f7542c3f43cf3e7f: Status 404 returned error can't find the container with id eb144c4129e29b4190a3aa909332f994a85f844d211a9709f7542c3f43cf3e7f
Oct 31 16:22:17.884633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.860248    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.917485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:17.917415    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:22:18.036591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.036556    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:22:18.037300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.037279    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:22:18.094509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.094053    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"]
Oct 31 16:22:18.132492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.112160    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:22:18.132492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.119330    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:22:18.132492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.132416    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"]
Oct 31 16:22:18.140631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.140558    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.146058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.146024    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"]
Oct 31 16:22:18.178946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.178618    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:18.178946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.178694    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:18.207639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.207609    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:22:18.233504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.224254    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.233504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.224339    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hmb\" (UniqueName: \"kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.233504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.224399    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.365481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.327628    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.365481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.327735    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hmb\" (UniqueName: \"kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.365481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.327821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.365481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.328857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.385173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.378766    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:22:18.467727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.441530    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hmb\" (UniqueName: \"kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.467727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.446540    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7fb4b73-sc5kd\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.520777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.520741    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:18.524930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.524848    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:22:18.568333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.568283    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv" event={"ID":"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0","Type":"ContainerStarted","Data":"eb144c4129e29b4190a3aa909332f994a85f844d211a9709f7542c3f43cf3e7f"}
Oct 31 16:22:18.863312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.863283    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:18.963942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.963906    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca\") pod \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") "
Oct 31 16:22:18.964145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.964029    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca\") pod \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") "
Oct 31 16:22:18.964145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.964073    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl\" (UniqueName: \"kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl\") pod \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\" (UID: \"87a645cb-02bf-47c5-8b80-b4882c50d9a2\") "
Oct 31 16:22:18.964283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.964245    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca" pod "87a645cb-02bf-47c5-8b80-b4882c50d9a2" (UID: "87a645cb-02bf-47c5-8b80-b4882c50d9a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:18.964778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.964752    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:18.966868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.966839    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl" pod "87a645cb-02bf-47c5-8b80-b4882c50d9a2" (UID: "87a645cb-02bf-47c5-8b80-b4882c50d9a2"). InnerVolumeSpecName "kube-api-access-fdrjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:18.967362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.967336    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca" pod "87a645cb-02bf-47c5-8b80-b4882c50d9a2" (UID: "87a645cb-02bf-47c5-8b80-b4882c50d9a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:18.976170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:18.976146    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"]
Oct 31 16:22:18.979317 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:18.979285    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8dbf549_6b29_49f0_9769_36c73c81e0a8.slice/crio-6d35b50c3bfd51bbd5e9b0017eace3a806fed539fd4c6e82036b540267cd9c76 WatchSource:0}: Error finding container 6d35b50c3bfd51bbd5e9b0017eace3a806fed539fd4c6e82036b540267cd9c76: Status 404 returned error can't find the container with id 6d35b50c3bfd51bbd5e9b0017eace3a806fed539fd4c6e82036b540267cd9c76
Oct 31 16:22:19.057505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.057343    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"]
Oct 31 16:22:19.064870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.064838    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5367cde-2wq9q"]
Oct 31 16:22:19.065787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.065763    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87a645cb-02bf-47c5-8b80-b4882c50d9a2-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:19.065787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.065784    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrjl\" (UniqueName: \"kubernetes.io/projected/87a645cb-02bf-47c5-8b80-b4882c50d9a2-kube-api-access-fdrjl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:19.179931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.179762    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:19.179931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.179817    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:19.594359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.594313    4519 generic.go:356] "Generic (PLEG): container finished" podID="09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" containerID="ed32cc9c7115a9d34961a8ae9b1198bab91335ba2b392308f3309fdebc35ec2a" exitCode=0
Oct 31 16:22:19.594766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.594577    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv" event={"ID":"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0","Type":"ContainerDied","Data":"ed32cc9c7115a9d34961a8ae9b1198bab91335ba2b392308f3309fdebc35ec2a"}
Oct 31 16:22:19.596703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.596677    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht" event={"ID":"87a645cb-02bf-47c5-8b80-b4882c50d9a2","Type":"ContainerDied","Data":"05493aafdcaa13593fab5132e5eccecff7a55efb646656ecf2cd64206020c989"}
Oct 31 16:22:19.596842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.596724    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05493aafdcaa13593fab5132e5eccecff7a55efb646656ecf2cd64206020c989"
Oct 31 16:22:19.596842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.596782    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"
Oct 31 16:22:19.602413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.602388    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd" event={"ID":"e8dbf549-6b29-49f0-9769-36c73c81e0a8","Type":"ContainerStarted","Data":"2b09c676b69675268bce11674130df08a652a484a283affd53418cf9014d69e4"}
Oct 31 16:22:19.602571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.602436    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd" event={"ID":"e8dbf549-6b29-49f0-9769-36c73c81e0a8","Type":"ContainerStarted","Data":"6d35b50c3bfd51bbd5e9b0017eace3a806fed539fd4c6e82036b540267cd9c76"}
Oct 31 16:22:19.648370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.648021    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"]
Oct 31 16:22:19.648824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.648807    4519 cpu_manager.go:414] "containerMap: removing container" podUID="87a645cb-02bf-47c5-8b80-b4882c50d9a2" containerName="revoke-keys"
Oct 31 16:22:19.648874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.648825    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a645cb-02bf-47c5-8b80-b4882c50d9a2" containerName="revoke-keys"
Oct 31 16:22:19.648980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.648971    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a645cb-02bf-47c5-8b80-b4882c50d9a2" containerName="revoke-keys"
Oct 31 16:22:19.652740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.652724    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.710120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.710085    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"]
Oct 31 16:22:19.783103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783065    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783110    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783133    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfr98\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783302    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783385    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783561 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783542    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783729    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.783854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.783839    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885180    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885347    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885533    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885591    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfr98\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885659    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885739    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.885858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.885793    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.886417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.886253    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.886417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.886336    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.887629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.886794    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.904038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.893444    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.904038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.893857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.907411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.907352    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.910364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.908993    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.953514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.948206    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.953514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.948362    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfr98\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.975046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.975009    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:19.981774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.981746    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:19.983028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.983008    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:19.983772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:19.983750    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:22:20.155443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.155394    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd" podStartSLOduration=3.155381033 podStartE2EDuration="3.155381033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:20.152831448 +0000 UTC m=+1183.852760757" watchObservedRunningTime="2026-10-31 16:22:20.155381033 +0000 UTC m=+1183.855310343"
Oct 31 16:22:20.385662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.385626    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"]
Oct 31 16:22:20.401832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.401790    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:20.401922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.401879    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:20.401957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.401921    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:20.443553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.443527    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e" path="/var/lib/kubelet/pods/3f3fcc06-6cb5-411e-bdac-9e27b2e0c67e/volumes"
Oct 31 16:22:20.629685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.629516    4519 generic.go:356] "Generic (PLEG): container finished" podID="e8dbf549-6b29-49f0-9769-36c73c81e0a8" containerID="2b09c676b69675268bce11674130df08a652a484a283affd53418cf9014d69e4" exitCode=0
Oct 31 16:22:20.630171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.629691    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd" event={"ID":"e8dbf549-6b29-49f0-9769-36c73c81e0a8","Type":"ContainerDied","Data":"2b09c676b69675268bce11674130df08a652a484a283affd53418cf9014d69e4"}
Oct 31 16:22:20.632904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.632804    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" event={"ID":"ee9601ce-9efe-4454-8065-20d09ba0056e","Type":"ContainerStarted","Data":"6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3"}
Oct 31 16:22:20.632904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.632884    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" event={"ID":"ee9601ce-9efe-4454-8065-20d09ba0056e","Type":"ContainerStarted","Data":"fb81457d39c64384560af5322cfab93d3e4c7dc85b1d24f2b963b7acf2595c2f"}
Oct 31 16:22:20.808632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.808604    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:20.853304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.853253    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podStartSLOduration=1.8532392720000002 podStartE2EDuration="1.853239272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:20.848018508 +0000 UTC m=+1184.547947821" watchObservedRunningTime="2026-10-31 16:22:20.853239272 +0000 UTC m=+1184.553168615"
Oct 31 16:22:20.903518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.903487    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5\" (UniqueName: \"kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5\") pod \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") "
Oct 31 16:22:20.903696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.903676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca\") pod \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") "
Oct 31 16:22:20.903816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.903800    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca\") pod \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\" (UID: \"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0\") "
Oct 31 16:22:20.904207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.904180    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca" pod "09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" (UID: "09f6b22a-0d41-4281-a055-d6b1ab7ff2b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:20.904508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.904488    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:20.905668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.905646    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5" pod "09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" (UID: "09f6b22a-0d41-4281-a055-d6b1ab7ff2b0"). InnerVolumeSpecName "kube-api-access-wzbg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:20.905757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.905687    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca" pod "09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" (UID: "09f6b22a-0d41-4281-a055-d6b1ab7ff2b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:20.978929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:20.977096    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:21.005534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.005495    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:21.005534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.005525    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbg5\" (UniqueName: \"kubernetes.io/projected/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0-kube-api-access-wzbg5\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:21.644474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.644430    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"
Oct 31 16:22:21.644923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.644492    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv" event={"ID":"09f6b22a-0d41-4281-a055-d6b1ab7ff2b0","Type":"ContainerDied","Data":"eb144c4129e29b4190a3aa909332f994a85f844d211a9709f7542c3f43cf3e7f"}
Oct 31 16:22:21.644923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.644539    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb144c4129e29b4190a3aa909332f994a85f844d211a9709f7542c3f43cf3e7f"
Oct 31 16:22:21.908945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.908840    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:21.919216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.919190    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca\") pod \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") "
Oct 31 16:22:21.919329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.919255    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb\" (UniqueName: \"kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb\") pod \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") "
Oct 31 16:22:21.919382 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.919345    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca\") pod \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\" (UID: \"e8dbf549-6b29-49f0-9769-36c73c81e0a8\") "
Oct 31 16:22:21.919610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.919579    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca" pod "e8dbf549-6b29-49f0-9769-36c73c81e0a8" (UID: "e8dbf549-6b29-49f0-9769-36c73c81e0a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:21.920070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.920048    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:21.921375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.921344    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca" pod "e8dbf549-6b29-49f0-9769-36c73c81e0a8" (UID: "e8dbf549-6b29-49f0-9769-36c73c81e0a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:21.921492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.921370    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb" pod "e8dbf549-6b29-49f0-9769-36c73c81e0a8" (UID: "e8dbf549-6b29-49f0-9769-36c73c81e0a8"). InnerVolumeSpecName "kube-api-access-b2hmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:21.991977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.991941    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 16:22:21.992136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:21.992008    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 16:22:22.021499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.021445    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2hmb\" (UniqueName: \"kubernetes.io/projected/e8dbf549-6b29-49f0-9769-36c73c81e0a8-kube-api-access-b2hmb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:22.021499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.021496    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8dbf549-6b29-49f0-9769-36c73c81e0a8-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:22.654306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.654273    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"
Oct 31 16:22:22.654770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.654308    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd" event={"ID":"e8dbf549-6b29-49f0-9769-36c73c81e0a8","Type":"ContainerDied","Data":"6d35b50c3bfd51bbd5e9b0017eace3a806fed539fd4c6e82036b540267cd9c76"}
Oct 31 16:22:22.654770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.654373    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d35b50c3bfd51bbd5e9b0017eace3a806fed539fd4c6e82036b540267cd9c76"
Oct 31 16:22:22.984480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.981512    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 16:22:22.984480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:22.981578    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 16:22:23.991359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:23.990812    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 16:22:23.991359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:23.990870    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 16:22:24.018794 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:24.018758    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:22:24.987480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:24.986878    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 16:22:24.987480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:24.986945    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 16:22:25.989479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:25.986819    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 16:22:25.989479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:25.986880    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 16:22:26.975815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:26.975775    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 16:22:26.976000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:26.975854    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 16:22:27.980196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:27.980167    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:27.980702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:27.980685    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:27.981235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:27.981216    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:22:33.063411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.063378    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"]
Oct 31 16:22:33.064083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.064068    4519 cpu_manager.go:414] "containerMap: removing container" podUID="09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" containerName="revoke-keys"
Oct 31 16:22:33.064140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.064084    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" containerName="revoke-keys"
Oct 31 16:22:33.064140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.064114    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e8dbf549-6b29-49f0-9769-36c73c81e0a8" containerName="revoke-keys"
Oct 31 16:22:33.064140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.064119    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8dbf549-6b29-49f0-9769-36c73c81e0a8" containerName="revoke-keys"
Oct 31 16:22:33.064239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.064229    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" containerName="revoke-keys"
Oct 31 16:22:33.064280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.064245    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8dbf549-6b29-49f0-9769-36c73c81e0a8" containerName="revoke-keys"
Oct 31 16:22:33.067994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.067978    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.116820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.116784    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.116990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.116834    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.117071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.117046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz8k\" (UniqueName: \"kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.121899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.121871    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"]
Oct 31 16:22:33.163245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.163213    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:22:33.164834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.164807    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:22:33.164834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.164816    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:22:33.165019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.164879    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:22:33.218633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.218597    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz8k\" (UniqueName: \"kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.218791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.218770    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.218879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.218794    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.219507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.219482    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.221429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.221410    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.275836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.275804    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz8k\" (UniqueName: \"kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k\") pod \"maas-api-revoke-sub-cc4d54e6-8h86c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.382238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.382157    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:33.757700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:33.757655    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"]
Oct 31 16:22:33.762007 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:33.761970    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c WatchSource:0}: Error finding container 5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c: Status 404 returned error can't find the container with id 5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c
Oct 31 16:22:34.776608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:34.776569    4519 generic.go:356] "Generic (PLEG): container finished" podID="71d0fa60-1708-423a-9e9c-818b8930977c" containerID="c2405d660b0d841dc208f1bd5d77f66dcb976ff17016ae80f7379a725b73bc4d" exitCode=0
Oct 31 16:22:34.776993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:34.776643    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c" event={"ID":"71d0fa60-1708-423a-9e9c-818b8930977c","Type":"ContainerDied","Data":"c2405d660b0d841dc208f1bd5d77f66dcb976ff17016ae80f7379a725b73bc4d"}
Oct 31 16:22:34.776993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:34.776690    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c" event={"ID":"71d0fa60-1708-423a-9e9c-818b8930977c","Type":"ContainerStarted","Data":"5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c"}
Oct 31 16:22:34.935976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:34.935894    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"]
Oct 31 16:22:34.941493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:34.941467    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.005612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.005573    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"]
Oct 31 16:22:35.053376 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.053333    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.053587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.053438    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.053587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.053528    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.053764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.053600    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.053764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.053681    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.053888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.053764    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgxm\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.054342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.054312    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.054749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.054729    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.054905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.054874    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156256    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156328    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156406    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156550    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156592    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156633    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156700    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156785    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156830    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156835    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgxm\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.156981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.156837    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.157208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.157182    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.157291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.157267    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.157488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.157434    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.159055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.159033    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.159153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.159105    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.217755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.217706    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.223870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.223834    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgxm\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.255967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.255932    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:35.710475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.710390    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"]
Oct 31 16:22:35.713718 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:35.713691    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6bc8de_16c6_4fd1_aba9_92773f73f00d.slice/crio-3e61a54d6406e2613b74eeb22ef2ee4782ae596944e2e8daf079e3f1b4a4356d WatchSource:0}: Error finding container 3e61a54d6406e2613b74eeb22ef2ee4782ae596944e2e8daf079e3f1b4a4356d: Status 404 returned error can't find the container with id 3e61a54d6406e2613b74eeb22ef2ee4782ae596944e2e8daf079e3f1b4a4356d
Oct 31 16:22:35.716619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.716579    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:35.716697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.716667    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:35.716732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.716711    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:35.787366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.787330    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" event={"ID":"bf6bc8de-16c6-4fd1-aba9-92773f73f00d","Type":"ContainerStarted","Data":"3e61a54d6406e2613b74eeb22ef2ee4782ae596944e2e8daf079e3f1b4a4356d"}
Oct 31 16:22:35.939686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.939654    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:35.946683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.946651    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"]
Oct 31 16:22:35.947564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.947544    4519 cpu_manager.go:414] "containerMap: removing container" podUID="71d0fa60-1708-423a-9e9c-818b8930977c" containerName="revoke-keys"
Oct 31 16:22:35.947564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.947564    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d0fa60-1708-423a-9e9c-818b8930977c" containerName="revoke-keys"
Oct 31 16:22:35.947740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.947729    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d0fa60-1708-423a-9e9c-818b8930977c" containerName="revoke-keys"
Oct 31 16:22:35.951244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.951226    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:35.980099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.980024    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca\") pod \"71d0fa60-1708-423a-9e9c-818b8930977c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") "
Oct 31 16:22:35.980241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.980114    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k\" (UniqueName: \"kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k\") pod \"71d0fa60-1708-423a-9e9c-818b8930977c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") "
Oct 31 16:22:35.980241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.980143    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca\") pod \"71d0fa60-1708-423a-9e9c-818b8930977c\" (UID: \"71d0fa60-1708-423a-9e9c-818b8930977c\") "
Oct 31 16:22:35.980606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.980579    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca" pod "71d0fa60-1708-423a-9e9c-818b8930977c" (UID: "71d0fa60-1708-423a-9e9c-818b8930977c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:35.980942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.980921    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:35.982268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.982244    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k" pod "71d0fa60-1708-423a-9e9c-818b8930977c" (UID: "71d0fa60-1708-423a-9e9c-818b8930977c"). InnerVolumeSpecName "kube-api-access-5zz8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:35.982268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:35.982257    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca" pod "71d0fa60-1708-423a-9e9c-818b8930977c" (UID: "71d0fa60-1708-423a-9e9c-818b8930977c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:36.036927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.036057    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"]
Oct 31 16:22:36.081985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.081943    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082005    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082052    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfzn\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082077    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082107    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082142    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082312    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082341    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082381    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.082728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082641    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz8k\" (UniqueName: \"kubernetes.io/projected/71d0fa60-1708-423a-9e9c-818b8930977c-kube-api-access-5zz8k\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:36.082728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.082660    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71d0fa60-1708-423a-9e9c-818b8930977c-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:36.183828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.183790    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.183838    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.183985    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184180    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184199    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184220    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184264    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfzn\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184288    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184319    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184359    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184480    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.184782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.184757    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.185094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.185068    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.185187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.185094    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.186351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.186323    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.186711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.186694    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.238990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.238922    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.239126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.239058    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfzn\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.265638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.265610    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:36.362094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.362057    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"]
Oct 31 16:22:36.369140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.369116    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.421022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.420991    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"]
Oct 31 16:22:36.502442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.502346    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.504541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.504505    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpszg\" (UniqueName: \"kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.504676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.504656    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.605946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.605905    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.606147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.606006    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpszg\" (UniqueName: \"kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.606147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.606099    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.607295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.607264    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.609431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.609405    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.652124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.652090    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"]
Oct 31 16:22:36.656360 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:36.656332    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3c224c_3bdd_410a_b0bc_4ff880c613a3.slice/crio-7e0f38d9b52dd7f7597c25fe8ddf5caed27df1f599cf5c52ce7e1b1cc737a89f WatchSource:0}: Error finding container 7e0f38d9b52dd7f7597c25fe8ddf5caed27df1f599cf5c52ce7e1b1cc737a89f: Status 404 returned error can't find the container with id 7e0f38d9b52dd7f7597c25fe8ddf5caed27df1f599cf5c52ce7e1b1cc737a89f
Oct 31 16:22:36.656565 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.656546    4519 scope.go:122] "RemoveContainer" containerID="e413e46a6faa2b85bdabfa6a49cc53f74d2605c76c1f056df90b7710c551efc7"
Oct 31 16:22:36.659123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.659089    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.659252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.659155    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.659252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.659185    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.659689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.659639    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpszg\" (UniqueName: \"kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg\") pod \"maas-api-revoke-sub-4519e810-7z7js\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.697475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.693128    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:36.823019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.812902    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" event={"ID":"bf6bc8de-16c6-4fd1-aba9-92773f73f00d","Type":"ContainerStarted","Data":"60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2"}
Oct 31 16:22:36.823019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.818421    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" event={"ID":"bc3c224c-3bdd-410a-b0bc-4ff880c613a3","Type":"ContainerStarted","Data":"f3ba42ee3f6666586be3ee7e3b5ea83be222bad5646e8602535957c6514a9c82"}
Oct 31 16:22:36.823019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.818514    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" event={"ID":"bc3c224c-3bdd-410a-b0bc-4ff880c613a3","Type":"ContainerStarted","Data":"7e0f38d9b52dd7f7597c25fe8ddf5caed27df1f599cf5c52ce7e1b1cc737a89f"}
Oct 31 16:22:36.823019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.820074    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"
Oct 31 16:22:36.823019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.820091    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c" event={"ID":"71d0fa60-1708-423a-9e9c-818b8930977c","Type":"ContainerDied","Data":"5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c"}
Oct 31 16:22:36.823019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.820189    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c"
Oct 31 16:22:36.924816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:36.924753    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podStartSLOduration=2.924735408 podStartE2EDuration="2.924735408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:36.923880931 +0000 UTC m=+1200.623810249" watchObservedRunningTime="2026-10-31 16:22:36.924735408 +0000 UTC m=+1200.624664719"
Oct 31 16:22:37.048560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.048442    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"]
Oct 31 16:22:37.053151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.053122    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d85db3d-vlwfd"]
Oct 31 16:22:37.103718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.103661    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podStartSLOduration=2.103647461 podStartE2EDuration="2.103647461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:37.101075562 +0000 UTC m=+1200.801004874" watchObservedRunningTime="2026-10-31 16:22:37.103647461 +0000 UTC m=+1200.803576772"
Oct 31 16:22:37.123916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.123893    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"]
Oct 31 16:22:37.126333 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:37.126304    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627bd569_d8c2_4590_a2ba_5721ab406251.slice/crio-09d09ff47ecdf21abdd7c23be76062a3505137986f893dbccef601774f44df8a WatchSource:0}: Error finding container 09d09ff47ecdf21abdd7c23be76062a3505137986f893dbccef601774f44df8a: Status 404 returned error can't find the container with id 09d09ff47ecdf21abdd7c23be76062a3505137986f893dbccef601774f44df8a
Oct 31 16:22:37.203259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.203223    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:22:37.256542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.256511    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:37.258553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.258522    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:37.258695 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.258583    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:37.279970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.271786    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:37.295789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.294594    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:37.295789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.294678    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:37.438332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.438301    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"]
Oct 31 16:22:37.444024 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.444000    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.528656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.525021    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"]
Oct 31 16:22:37.528656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.526400    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.528656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.526507    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.528656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.526687    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxnph\" (UniqueName: \"kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.629407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.627798    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.629407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.627894    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.629407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.627979    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnph\" (UniqueName: \"kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.629407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.629186    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.634629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.634601    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.711505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.711265    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnph\" (UniqueName: \"kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph\") pod \"maas-api-revoke-sub-a04b8bdf-xqkns\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.759030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.758961    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:37.851423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.851326    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js" event={"ID":"627bd569-d8c2-4590-a2ba-5721ab406251","Type":"ContainerStarted","Data":"85313d0d306020460b0263740ec98b42e400ea2029e6a028c322895532b16f1d"}
Oct 31 16:22:37.851423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.851415    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js" event={"ID":"627bd569-d8c2-4590-a2ba-5721ab406251","Type":"ContainerStarted","Data":"09d09ff47ecdf21abdd7c23be76062a3505137986f893dbccef601774f44df8a"}
Oct 31 16:22:37.987475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:37.986310    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js" podStartSLOduration=1.986290267 podStartE2EDuration="1.986290267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:37.980963826 +0000 UTC m=+1201.680893138" watchObservedRunningTime="2026-10-31 16:22:37.986290267 +0000 UTC m=+1201.686219578"
Oct 31 16:22:38.372525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:38.337317    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:38.372525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:38.337378    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:38.386405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:38.386362    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:38.410510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:38.386646    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:38.600479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:38.600383    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caefea16-9a99-446b-9d95-920bc23c5580" path="/var/lib/kubelet/pods/caefea16-9a99-446b-9d95-920bc23c5580/volumes"
Oct 31 16:22:39.081981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.078928    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:39.148489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.133399    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.148489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.136974    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"]
Oct 31 16:22:39.155437 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:39.155394    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a50218e_f83c_4f38_9c36_bf5f9a61a791.slice/crio-2e389b45bb8b10e3b67559d32e902a31c27c8c6c2af5f4d4ad3b43c2c7a30de4 WatchSource:0}: Error finding container 2e389b45bb8b10e3b67559d32e902a31c27c8c6c2af5f4d4ad3b43c2c7a30de4: Status 404 returned error can't find the container with id 2e389b45bb8b10e3b67559d32e902a31c27c8c6c2af5f4d4ad3b43c2c7a30de4
Oct 31 16:22:39.225716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.225657    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.266311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.266266    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:39.266546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.266334    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:39.267303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.267227    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:39.267303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.267267    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:39.282073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.282024    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.376734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.376636    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.416691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.416639    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.503669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.503609    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.782265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:39.778540    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.163480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.110720    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:40.163480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.111114    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.163480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.111395    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:22:40.207890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.183496    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:22:40.207890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.184797    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:22:40.207890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.185232    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:22:40.249481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.246579    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:40.249481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.246647    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.281486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.280352    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:40.281486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.280414    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:40.281486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.280780    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:40.281486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.280814    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:40.310538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.285743    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.337478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.311229    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:22:40.337478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.296575    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.420179    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.420760    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.421109    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.421478    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.421558    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.421948    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:22:40.460481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.422631    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:22:40.508822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.508767    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:22:40.527852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.516624    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.739840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.716315    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"]
Oct 31 16:22:40.740219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.740192    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:22:40.943479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:40.941739    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"]
Oct 31 16:22:41.018539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.018194    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns" event={"ID":"9a50218e-f83c-4f38-9c36-bf5f9a61a791","Type":"ContainerStarted","Data":"64f40d3c9161a6cfe0990f75c951697520c720c188d7c8782bc4fd14c369a63d"}
Oct 31 16:22:41.018539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.018263    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns" event={"ID":"9a50218e-f83c-4f38-9c36-bf5f9a61a791","Type":"ContainerStarted","Data":"2e389b45bb8b10e3b67559d32e902a31c27c8c6c2af5f4d4ad3b43c2c7a30de4"}
Oct 31 16:22:41.067652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.061326    4519 generic.go:356] "Generic (PLEG): container finished" podID="627bd569-d8c2-4590-a2ba-5721ab406251" containerID="85313d0d306020460b0263740ec98b42e400ea2029e6a028c322895532b16f1d" exitCode=0
Oct 31 16:22:41.067652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.061633    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js" event={"ID":"627bd569-d8c2-4590-a2ba-5721ab406251","Type":"ContainerDied","Data":"85313d0d306020460b0263740ec98b42e400ea2029e6a028c322895532b16f1d"}
Oct 31 16:22:41.283537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.281528    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:41.283537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.281930    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.283537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.282301    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:22:41.303652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.298552    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.303652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.299042    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:22:41.342511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.326595    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:22:41.342511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.327511    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.342511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.328329    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:22:41.351142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.350531    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:41.351142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.350609    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:41.401625 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.401483    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:41.401625 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.401559    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:41.417037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.416983    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.445012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.444963    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.478811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.478751    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.529518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.524547    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.529518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.504999    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:22:41.529518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.505046    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:22:41.530578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.504932    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa900bc-45n8b"]
Oct 31 16:22:41.569231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.569138    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:22:41.569883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.568896    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.570491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.570471    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:22:41.680150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:41.680106    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 16:22:42.257616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.257515    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2959b67-6n8gs"]
Oct 31 16:22:42.318496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.258237    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:42.318496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.258290    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:42.318496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.273564    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:42.318496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.273622    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:42.318496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.312650    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.342491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.334145    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.359690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.308925    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.370552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.366113    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.370552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.367164    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.444124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.443300    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.511594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.502822    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.699804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.599995    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.699804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.601493    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.699804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.602044    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.699804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.602386    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.699804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.607953    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.885741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.885627    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:43.015241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.015189    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.238715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:42.949169    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 31 16:22:43.383223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.313125    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.416544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.331538    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:43.416832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.416798    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.417043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.331636    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:43.417195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.417171    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:43.430651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.357972    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:22:43.446590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.446536    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:43.446907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.446850    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:43.517538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.499335    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:22:43.617371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.517604    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:22:43.755496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.744288    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.069239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.518742    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.223069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.729537    4519 generic.go:356] "Generic (PLEG): container finished" podID="9a50218e-f83c-4f38-9c36-bf5f9a61a791" containerID="64f40d3c9161a6cfe0990f75c951697520c720c188d7c8782bc4fd14c369a63d" exitCode=0
Oct 31 16:22:44.223557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.733303    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.223789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.733380    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.223986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.733824    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.224161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.733867    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.224337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.733888    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 31 16:22:44.224514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.733931    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.224706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.735156    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx" podUID="602afcaa-d57d-4bde-aadb-646f36b8733f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.225009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.518708    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.225200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:43.797518    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578aec0c-f3a9-471c-b7b8-98c35fb3bab3" path="/var/lib/kubelet/pods/578aec0c-f3a9-471c-b7b8-98c35fb3bab3/volumes"
Oct 31 16:22:44.300024 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.298543    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:44.300024 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.298600    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:44.300339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.300309    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:44.300483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.300442    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:44.413263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.413166    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.415127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.415095    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/health\": context deadline exceeded"
Oct 31 16:22:44.462510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.458625    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.539476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.529912    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b4ad60-92dc-4084-8fe5-d047b6f3aa89" path="/var/lib/kubelet/pods/80b4ad60-92dc-4084-8fe5-d047b6f3aa89/volumes"
Oct 31 16:22:44.539476 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:44.537772    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.092s"
Oct 31 16:22:44.601591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.565860    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns" podStartSLOduration=7.5658433800000005 podStartE2EDuration="7.56584338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:42.05126653 +0000 UTC m=+1205.751195844" watchObservedRunningTime="2026-10-31 16:22:44.56584338 +0000 UTC m=+1208.265772691"
Oct 31 16:22:44.759576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.754989    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:22:44.759918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.759901    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:22:44.760033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760024    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:22:44.760132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760124    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:22:44.760232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760224    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:22:44.760318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760303    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns" event={"ID":"9a50218e-f83c-4f38-9c36-bf5f9a61a791","Type":"ContainerDied","Data":"64f40d3c9161a6cfe0990f75c951697520c720c188d7c8782bc4fd14c369a63d"}
Oct 31 16:22:44.760444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760433    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:22:44.760559 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760550    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:22:44.760659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760651    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:22:44.760758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760750    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:22:44.760897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.760887    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:22:44.825498 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:44.824823    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:22:44.825498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.825126    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:22:44.826481 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:44.826332    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:22:44.842073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:44.829739    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:22:45.059751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.059722    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:45.238616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.238583    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"
Oct 31 16:22:45.238616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.238591    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js" event={"ID":"627bd569-d8c2-4590-a2ba-5721ab406251","Type":"ContainerDied","Data":"09d09ff47ecdf21abdd7c23be76062a3505137986f893dbccef601774f44df8a"}
Oct 31 16:22:45.238883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.238673    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d09ff47ecdf21abdd7c23be76062a3505137986f893dbccef601774f44df8a"
Oct 31 16:22:45.256766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.256631    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:45.256766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.256684    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:45.258521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.258373    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca\") pod \"627bd569-d8c2-4590-a2ba-5721ab406251\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") "
Oct 31 16:22:45.258671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.258598    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca\") pod \"627bd569-d8c2-4590-a2ba-5721ab406251\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") "
Oct 31 16:22:45.258671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.258638    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg\" (UniqueName: \"kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg\") pod \"627bd569-d8c2-4590-a2ba-5721ab406251\" (UID: \"627bd569-d8c2-4590-a2ba-5721ab406251\") "
Oct 31 16:22:45.259065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.259038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca" pod "627bd569-d8c2-4590-a2ba-5721ab406251" (UID: "627bd569-d8c2-4590-a2ba-5721ab406251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:45.259186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.259167    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:45.260899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.260873    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca" pod "627bd569-d8c2-4590-a2ba-5721ab406251" (UID: "627bd569-d8c2-4590-a2ba-5721ab406251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:45.261649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.261617    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg" pod "627bd569-d8c2-4590-a2ba-5721ab406251" (UID: "627bd569-d8c2-4590-a2ba-5721ab406251"). InnerVolumeSpecName "kube-api-access-vpszg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:45.289739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.272902    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:45.289739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.272956    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:45.360809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.360772    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpszg\" (UniqueName: \"kubernetes.io/projected/627bd569-d8c2-4590-a2ba-5721ab406251-kube-api-access-vpszg\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:45.360809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.360808    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627bd569-d8c2-4590-a2ba-5721ab406251-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:45.622653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.622626    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:45.673528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.670109    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca\") pod \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") "
Oct 31 16:22:45.674145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.674123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph\" (UniqueName: \"kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph\") pod \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") "
Oct 31 16:22:45.680599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.680575    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca\") pod \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\" (UID: \"9a50218e-f83c-4f38-9c36-bf5f9a61a791\") "
Oct 31 16:22:45.689489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.688692    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca" pod "9a50218e-f83c-4f38-9c36-bf5f9a61a791" (UID: "9a50218e-f83c-4f38-9c36-bf5f9a61a791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:45.690354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.690334    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:45.706239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.706199    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca" pod "9a50218e-f83c-4f38-9c36-bf5f9a61a791" (UID: "9a50218e-f83c-4f38-9c36-bf5f9a61a791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:45.709407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.709363    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph" pod "9a50218e-f83c-4f38-9c36-bf5f9a61a791" (UID: "9a50218e-f83c-4f38-9c36-bf5f9a61a791"). InnerVolumeSpecName "kube-api-access-xxnph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:45.792890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.792837    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxnph\" (UniqueName: \"kubernetes.io/projected/9a50218e-f83c-4f38-9c36-bf5f9a61a791-kube-api-access-xxnph\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:45.792890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:45.792880    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a50218e-f83c-4f38-9c36-bf5f9a61a791-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:46.251328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.251295    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns" event={"ID":"9a50218e-f83c-4f38-9c36-bf5f9a61a791","Type":"ContainerDied","Data":"2e389b45bb8b10e3b67559d32e902a31c27c8c6c2af5f4d4ad3b43c2c7a30de4"}
Oct 31 16:22:46.251533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.251345    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e389b45bb8b10e3b67559d32e902a31c27c8c6c2af5f4d4ad3b43c2c7a30de4"
Oct 31 16:22:46.251533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.251349    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"
Oct 31 16:22:46.256559 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.256533    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 16:22:46.256695 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.256586    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 16:22:46.290549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.289930    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:46.290549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.289985    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:46.948598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.948556    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"]
Oct 31 16:22:46.949279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.949262    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9a50218e-f83c-4f38-9c36-bf5f9a61a791" containerName="revoke-keys"
Oct 31 16:22:46.949279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.949279    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a50218e-f83c-4f38-9c36-bf5f9a61a791" containerName="revoke-keys"
Oct 31 16:22:46.949446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.949309    4519 cpu_manager.go:414] "containerMap: removing container" podUID="627bd569-d8c2-4590-a2ba-5721ab406251" containerName="revoke-keys"
Oct 31 16:22:46.949446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.949314    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627bd569-d8c2-4590-a2ba-5721ab406251" containerName="revoke-keys"
Oct 31 16:22:46.949446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.949409    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="627bd569-d8c2-4590-a2ba-5721ab406251" containerName="revoke-keys"
Oct 31 16:22:46.949446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.949425    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a50218e-f83c-4f38-9c36-bf5f9a61a791" containerName="revoke-keys"
Oct 31 16:22:46.953639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.953610    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:46.991145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:46.988911    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"]
Oct 31 16:22:47.003721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.003691    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:22:47.003895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.003761    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:22:47.003895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.003861    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:22:47.003895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.003695    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:22:47.123734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.123704    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.123927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.123883    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.123999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.123985    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthl2\" (UniqueName: \"kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.225523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.225490    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthl2\" (UniqueName: \"kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.225707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.225691    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.225814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.225799    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.226507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.226484    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.228054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.228036    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.261236 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.261184    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:47.262332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.262309    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:47.263248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.263230    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:22:47.266593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.266569    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 16:22:47.266675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.266614    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 16:22:47.279260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.279232    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthl2\" (UniqueName: \"kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2\") pod \"maas-api-revoke-sub-def6daf6-8jj8l\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.571734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.571536    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:47.778099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.778064    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"]
Oct 31 16:22:47.783875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.783851    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:47.839067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.838503    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"]
Oct 31 16:22:47.940575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.940532    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkff8\" (UniqueName: \"kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:47.940769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.940646    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:47.940769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.940728    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:47.980582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:47.980550    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"]
Oct 31 16:22:48.049196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.049018    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkff8\" (UniqueName: \"kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.049196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.049078    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.049196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.049147    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.050020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.049983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.052588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.052566    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.103572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.103504    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkff8\" (UniqueName: \"kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8\") pod \"maas-api-revoke-sub-ecde7be7-6mw7z\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.270593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.270562    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:48.270912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.270884    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:48.271756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.271736    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:22:48.273508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.273486    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l" event={"ID":"9467039d-55c5-4574-91d9-cd1023b29d0d","Type":"ContainerStarted","Data":"1a9c654160d5442f2510aeba91cb1c7cacca56944f1831c42c20e26036a10f50"}
Oct 31 16:22:48.401919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.401844    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:48.577322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.577284    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"]
Oct 31 16:22:48.592248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.591562    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.634088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.634058    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"]
Oct 31 16:22:48.657696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.657605    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhl8\" (UniqueName: \"kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.657855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.657769    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.657855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.657836    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.759021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.758987    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.759210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.759200    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhl8\" (UniqueName: \"kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.759284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.759260    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.759735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.759708    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.761918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.761899    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.792080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.792050    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"]
Oct 31 16:22:48.813505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.813478    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhl8\" (UniqueName: \"kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8\") pod \"maas-api-revoke-sub-36ef8190-sfj8g\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:48.913849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:48.913772    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:49.283527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:49.283486    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z" event={"ID":"1ef8b50f-073d-4e5a-933d-2321cfd82d98","Type":"ContainerStarted","Data":"6ce20f192379ad43dd6e79cbd21bab48fcb80c32142c78cc17a2b0e0897e789b"}
Oct 31 16:22:49.284001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:49.283538    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z" event={"ID":"1ef8b50f-073d-4e5a-933d-2321cfd82d98","Type":"ContainerStarted","Data":"61ebcb551f466676198aead4ab207544d4f11d24b0afabd96b851c780d1e529d"}
Oct 31 16:22:49.285323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:49.285296    4519 generic.go:356] "Generic (PLEG): container finished" podID="9467039d-55c5-4574-91d9-cd1023b29d0d" containerID="7a77e10cec34c42c9257568a38b0a70ba0bb1291939e7195dbca8098732457db" exitCode=0
Oct 31 16:22:49.285440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:49.285370    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l" event={"ID":"9467039d-55c5-4574-91d9-cd1023b29d0d","Type":"ContainerDied","Data":"7a77e10cec34c42c9257568a38b0a70ba0bb1291939e7195dbca8098732457db"}
Oct 31 16:22:49.340521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:49.340483    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"]
Oct 31 16:22:49.345324 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:49.345286    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bff5c0_a483_495c_b4c0_05159a8f4e73.slice/crio-226ad6072dc30bd2a6370ee3f5d835113cbec303a4587732e4b187733ec2165c WatchSource:0}: Error finding container 226ad6072dc30bd2a6370ee3f5d835113cbec303a4587732e4b187733ec2165c: Status 404 returned error can't find the container with id 226ad6072dc30bd2a6370ee3f5d835113cbec303a4587732e4b187733ec2165c
Oct 31 16:22:49.396773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:49.396726    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z" podStartSLOduration=2.396713542 podStartE2EDuration="2.396713542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:49.392641836 +0000 UTC m=+1213.092571149" watchObservedRunningTime="2026-10-31 16:22:49.396713542 +0000 UTC m=+1213.096642852"
Oct 31 16:22:50.309596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.309433    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g" event={"ID":"a4bff5c0-a483-495c-b4c0-05159a8f4e73","Type":"ContainerStarted","Data":"2f73421dfbbf72b0954d888800c058e3dfcf83ea1abf154241790e7db3395cf2"}
Oct 31 16:22:50.310082 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.309676    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g" event={"ID":"a4bff5c0-a483-495c-b4c0-05159a8f4e73","Type":"ContainerStarted","Data":"226ad6072dc30bd2a6370ee3f5d835113cbec303a4587732e4b187733ec2165c"}
Oct 31 16:22:50.311522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.311444    4519 generic.go:356] "Generic (PLEG): container finished" podID="1ef8b50f-073d-4e5a-933d-2321cfd82d98" containerID="6ce20f192379ad43dd6e79cbd21bab48fcb80c32142c78cc17a2b0e0897e789b" exitCode=0
Oct 31 16:22:50.311707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.311529    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z" event={"ID":"1ef8b50f-073d-4e5a-933d-2321cfd82d98","Type":"ContainerDied","Data":"6ce20f192379ad43dd6e79cbd21bab48fcb80c32142c78cc17a2b0e0897e789b"}
Oct 31 16:22:50.426344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.426271    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g" podStartSLOduration=2.426253056 podStartE2EDuration="2.426253056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:50.42091409 +0000 UTC m=+1214.120843402" watchObservedRunningTime="2026-10-31 16:22:50.426253056 +0000 UTC m=+1214.126182368"
Oct 31 16:22:50.556621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.556595    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:50.584506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.584408    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca\") pod \"9467039d-55c5-4574-91d9-cd1023b29d0d\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") "
Oct 31 16:22:50.584638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.584507    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2\" (UniqueName: \"kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2\") pod \"9467039d-55c5-4574-91d9-cd1023b29d0d\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") "
Oct 31 16:22:50.584638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.584558    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca\") pod \"9467039d-55c5-4574-91d9-cd1023b29d0d\" (UID: \"9467039d-55c5-4574-91d9-cd1023b29d0d\") "
Oct 31 16:22:50.584805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.584779    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca" pod "9467039d-55c5-4574-91d9-cd1023b29d0d" (UID: "9467039d-55c5-4574-91d9-cd1023b29d0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:50.585231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.585206    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:50.586767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.586738    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca" pod "9467039d-55c5-4574-91d9-cd1023b29d0d" (UID: "9467039d-55c5-4574-91d9-cd1023b29d0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:50.586875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.586847    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2" pod "9467039d-55c5-4574-91d9-cd1023b29d0d" (UID: "9467039d-55c5-4574-91d9-cd1023b29d0d"). InnerVolumeSpecName "kube-api-access-hthl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:50.687113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.687077    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hthl2\" (UniqueName: \"kubernetes.io/projected/9467039d-55c5-4574-91d9-cd1023b29d0d-kube-api-access-hthl2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:50.687113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:50.687116    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9467039d-55c5-4574-91d9-cd1023b29d0d-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:51.266058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.266020    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"]
Oct 31 16:22:51.267091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.267062    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9467039d-55c5-4574-91d9-cd1023b29d0d" containerName="revoke-keys"
Oct 31 16:22:51.267091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.267087    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9467039d-55c5-4574-91d9-cd1023b29d0d" containerName="revoke-keys"
Oct 31 16:22:51.267330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.267304    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9467039d-55c5-4574-91d9-cd1023b29d0d" containerName="revoke-keys"
Oct 31 16:22:51.271353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.271334    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.295330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.295304    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.295483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.295449    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.295567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.295545    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbgm\" (UniqueName: \"kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.323356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.323323    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"]
Oct 31 16:22:51.326633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.326606    4519 generic.go:356] "Generic (PLEG): container finished" podID="a4bff5c0-a483-495c-b4c0-05159a8f4e73" containerID="2f73421dfbbf72b0954d888800c058e3dfcf83ea1abf154241790e7db3395cf2" exitCode=0
Oct 31 16:22:51.326765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.326703    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g" event={"ID":"a4bff5c0-a483-495c-b4c0-05159a8f4e73","Type":"ContainerDied","Data":"2f73421dfbbf72b0954d888800c058e3dfcf83ea1abf154241790e7db3395cf2"}
Oct 31 16:22:51.328381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.328366    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"
Oct 31 16:22:51.328505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.328409    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l" event={"ID":"9467039d-55c5-4574-91d9-cd1023b29d0d","Type":"ContainerDied","Data":"1a9c654160d5442f2510aeba91cb1c7cacca56944f1831c42c20e26036a10f50"}
Oct 31 16:22:51.328505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.328450    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9c654160d5442f2510aeba91cb1c7cacca56944f1831c42c20e26036a10f50"
Oct 31 16:22:51.396857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.396825    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.397020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.396926    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.397020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.396964    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbgm\" (UniqueName: \"kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.398804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.398772    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.399897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.399873    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.452305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.452239    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbgm\" (UniqueName: \"kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm\") pod \"maas-api-revoke-sub-18571a62-jgzzd\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.500088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.500063    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:51.587023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.586944    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:51.599651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.599619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca\") pod \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") "
Oct 31 16:22:51.599800 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.599667    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca\") pod \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") "
Oct 31 16:22:51.599953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.599885    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8\" (UniqueName: \"kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8\") pod \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\" (UID: \"1ef8b50f-073d-4e5a-933d-2321cfd82d98\") "
Oct 31 16:22:51.600050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.599954    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca" pod "1ef8b50f-073d-4e5a-933d-2321cfd82d98" (UID: "1ef8b50f-073d-4e5a-933d-2321cfd82d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:51.600612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.600586    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:51.602370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.602314    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8" pod "1ef8b50f-073d-4e5a-933d-2321cfd82d98" (UID: "1ef8b50f-073d-4e5a-933d-2321cfd82d98"). InnerVolumeSpecName "kube-api-access-qkff8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:51.606849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.606824    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca" pod "1ef8b50f-073d-4e5a-933d-2321cfd82d98" (UID: "1ef8b50f-073d-4e5a-933d-2321cfd82d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:51.702240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.702207    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkff8\" (UniqueName: \"kubernetes.io/projected/1ef8b50f-073d-4e5a-933d-2321cfd82d98-kube-api-access-qkff8\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:51.702240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.702236    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef8b50f-073d-4e5a-933d-2321cfd82d98-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:51.965849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:51.965821    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"]
Oct 31 16:22:51.969536 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:51.969499    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa8ea29_d370_422f_869e_98fb3df77078.slice/crio-213d227afb46e20d40c253fc8cd83624298fb639c63df409fa4a5014b19406df WatchSource:0}: Error finding container 213d227afb46e20d40c253fc8cd83624298fb639c63df409fa4a5014b19406df: Status 404 returned error can't find the container with id 213d227afb46e20d40c253fc8cd83624298fb639c63df409fa4a5014b19406df
Oct 31 16:22:52.337576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.337537    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd" event={"ID":"4fa8ea29-d370-422f-869e-98fb3df77078","Type":"ContainerStarted","Data":"98e7f2a80e61747f65b16ca7da26e4cd371660631b920a8168de7c5d1a2e79b5"}
Oct 31 16:22:52.338044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.337603    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd" event={"ID":"4fa8ea29-d370-422f-869e-98fb3df77078","Type":"ContainerStarted","Data":"213d227afb46e20d40c253fc8cd83624298fb639c63df409fa4a5014b19406df"}
Oct 31 16:22:52.338991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.338968    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"
Oct 31 16:22:52.339090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.339011    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z" event={"ID":"1ef8b50f-073d-4e5a-933d-2321cfd82d98","Type":"ContainerDied","Data":"61ebcb551f466676198aead4ab207544d4f11d24b0afabd96b851c780d1e529d"}
Oct 31 16:22:52.339090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.339057    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ebcb551f466676198aead4ab207544d4f11d24b0afabd96b851c780d1e529d"
Oct 31 16:22:52.450036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.449976    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd" podStartSLOduration=1.449961676 podStartE2EDuration="1.449961676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:52.445915057 +0000 UTC m=+1216.145844371" watchObservedRunningTime="2026-10-31 16:22:52.449961676 +0000 UTC m=+1216.149890986"
Oct 31 16:22:52.508938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.508913    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:52.616089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.616004    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca\") pod \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") "
Oct 31 16:22:52.616241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.616092    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8\" (UniqueName: \"kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8\") pod \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") "
Oct 31 16:22:52.616280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.616256    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca\") pod \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\" (UID: \"a4bff5c0-a483-495c-b4c0-05159a8f4e73\") "
Oct 31 16:22:52.616708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.616676    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca" pod "a4bff5c0-a483-495c-b4c0-05159a8f4e73" (UID: "a4bff5c0-a483-495c-b4c0-05159a8f4e73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:52.616955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.616935    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:52.618248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.618228    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca" pod "a4bff5c0-a483-495c-b4c0-05159a8f4e73" (UID: "a4bff5c0-a483-495c-b4c0-05159a8f4e73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:52.619247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.619222    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8" pod "a4bff5c0-a483-495c-b4c0-05159a8f4e73" (UID: "a4bff5c0-a483-495c-b4c0-05159a8f4e73"). InnerVolumeSpecName "kube-api-access-shhl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:52.718395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.718364    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4bff5c0-a483-495c-b4c0-05159a8f4e73-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:52.718395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:52.718393    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhl8\" (UniqueName: \"kubernetes.io/projected/a4bff5c0-a483-495c-b4c0-05159a8f4e73-kube-api-access-shhl8\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:53.044914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.044881    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"]
Oct 31 16:22:53.049433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.049411    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee10422f-gvc6g"]
Oct 31 16:22:53.349524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.349164    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"
Oct 31 16:22:53.349524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.349178    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g" event={"ID":"a4bff5c0-a483-495c-b4c0-05159a8f4e73","Type":"ContainerDied","Data":"226ad6072dc30bd2a6370ee3f5d835113cbec303a4587732e4b187733ec2165c"}
Oct 31 16:22:53.349524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.349236    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="226ad6072dc30bd2a6370ee3f5d835113cbec303a4587732e4b187733ec2165c"
Oct 31 16:22:53.351175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.351150    4519 generic.go:356] "Generic (PLEG): container finished" podID="4fa8ea29-d370-422f-869e-98fb3df77078" containerID="98e7f2a80e61747f65b16ca7da26e4cd371660631b920a8168de7c5d1a2e79b5" exitCode=0
Oct 31 16:22:53.351296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.351205    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd" event={"ID":"4fa8ea29-d370-422f-869e-98fb3df77078","Type":"ContainerDied","Data":"98e7f2a80e61747f65b16ca7da26e4cd371660631b920a8168de7c5d1a2e79b5"}
Oct 31 16:22:53.867023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.866984    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"]
Oct 31 16:22:53.867782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.867761    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1ef8b50f-073d-4e5a-933d-2321cfd82d98" containerName="revoke-keys"
Oct 31 16:22:53.867782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.867782    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef8b50f-073d-4e5a-933d-2321cfd82d98" containerName="revoke-keys"
Oct 31 16:22:53.867957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.867833    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a4bff5c0-a483-495c-b4c0-05159a8f4e73" containerName="revoke-keys"
Oct 31 16:22:53.867957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.867841    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bff5c0-a483-495c-b4c0-05159a8f4e73" containerName="revoke-keys"
Oct 31 16:22:53.868066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.868027    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef8b50f-073d-4e5a-933d-2321cfd82d98" containerName="revoke-keys"
Oct 31 16:22:53.868066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.868050    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bff5c0-a483-495c-b4c0-05159a8f4e73" containerName="revoke-keys"
Oct 31 16:22:53.871796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.871776    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:53.923792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.923765    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"]
Oct 31 16:22:53.935584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.935558    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:53.935740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.935723    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:53.935822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:53.935797    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gpz\" (UniqueName: \"kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.004931 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:54.004895    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:22:54.036985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.036953    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.036985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.036984    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gpz\" (UniqueName: \"kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.037204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.037187    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.037728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.037710    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.039463 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.039434    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.090478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.090439    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gpz\" (UniqueName: \"kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz\") pod \"maas-api-revoke-sub-8e66d246-q7qmq\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.187069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.186996    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:54.450048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.449952    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6" path="/var/lib/kubelet/pods/75f6092f-7ee0-4bb7-8fe2-b81c4a79d7d6/volumes"
Oct 31 16:22:54.538129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.538100    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:54.564946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.564920    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"]
Oct 31 16:22:54.568523 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:54.568499    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d5cddc_a102_4ba4_a58c_8b1d531865ef.slice/crio-5ec8b6fe3a817812a0febf188cd171f4834ae4dfd97353b4104feca4baaf283c WatchSource:0}: Error finding container 5ec8b6fe3a817812a0febf188cd171f4834ae4dfd97353b4104feca4baaf283c: Status 404 returned error can't find the container with id 5ec8b6fe3a817812a0febf188cd171f4834ae4dfd97353b4104feca4baaf283c
Oct 31 16:22:54.654474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.654420    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca\") pod \"4fa8ea29-d370-422f-869e-98fb3df77078\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") "
Oct 31 16:22:54.654629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.654532    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca\") pod \"4fa8ea29-d370-422f-869e-98fb3df77078\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") "
Oct 31 16:22:54.654843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.654824    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm\" (UniqueName: \"kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm\") pod \"4fa8ea29-d370-422f-869e-98fb3df77078\" (UID: \"4fa8ea29-d370-422f-869e-98fb3df77078\") "
Oct 31 16:22:54.656101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.656073    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca" pod "4fa8ea29-d370-422f-869e-98fb3df77078" (UID: "4fa8ea29-d370-422f-869e-98fb3df77078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:54.657642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.657602    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm" pod "4fa8ea29-d370-422f-869e-98fb3df77078" (UID: "4fa8ea29-d370-422f-869e-98fb3df77078"). InnerVolumeSpecName "kube-api-access-lhbgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:54.658485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.658422    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca" pod "4fa8ea29-d370-422f-869e-98fb3df77078" (UID: "4fa8ea29-d370-422f-869e-98fb3df77078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:54.757123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.757094    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhbgm\" (UniqueName: \"kubernetes.io/projected/4fa8ea29-d370-422f-869e-98fb3df77078-kube-api-access-lhbgm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:54.757123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.757120    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:54.757323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:54.757133    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa8ea29-d370-422f-869e-98fb3df77078-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:54.924840 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:54.924764    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:22:55.047769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.047673    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"]
Oct 31 16:22:55.052262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.052232    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b10338-ttldl"]
Oct 31 16:22:55.370645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.370569    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"
Oct 31 16:22:55.370799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.370576    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd" event={"ID":"4fa8ea29-d370-422f-869e-98fb3df77078","Type":"ContainerDied","Data":"213d227afb46e20d40c253fc8cd83624298fb639c63df409fa4a5014b19406df"}
Oct 31 16:22:55.370799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.370757    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="213d227afb46e20d40c253fc8cd83624298fb639c63df409fa4a5014b19406df"
Oct 31 16:22:55.372601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.372570    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq" event={"ID":"07d5cddc-a102-4ba4-a58c-8b1d531865ef","Type":"ContainerStarted","Data":"a28e9d8344099c25df70b729bc23899ac40a8e1193a7269d6b371e549bd62045"}
Oct 31 16:22:55.372737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.372621    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq" event={"ID":"07d5cddc-a102-4ba4-a58c-8b1d531865ef","Type":"ContainerStarted","Data":"5ec8b6fe3a817812a0febf188cd171f4834ae4dfd97353b4104feca4baaf283c"}
Oct 31 16:22:55.481689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:55.481632    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq" podStartSLOduration=2.481548068 podStartE2EDuration="2.481548068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:55.479107927 +0000 UTC m=+1219.179037237" watchObservedRunningTime="2026-10-31 16:22:55.481548068 +0000 UTC m=+1219.181477376"
Oct 31 16:22:56.045340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.045304    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"]
Oct 31 16:22:56.049804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.049777    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26a68870-b9gkx"]
Oct 31 16:22:56.382317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.382236    4519 generic.go:356] "Generic (PLEG): container finished" podID="07d5cddc-a102-4ba4-a58c-8b1d531865ef" containerID="a28e9d8344099c25df70b729bc23899ac40a8e1193a7269d6b371e549bd62045" exitCode=0
Oct 31 16:22:56.382522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.382321    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq" event={"ID":"07d5cddc-a102-4ba4-a58c-8b1d531865ef","Type":"ContainerDied","Data":"a28e9d8344099c25df70b729bc23899ac40a8e1193a7269d6b371e549bd62045"}
Oct 31 16:22:56.441353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.441311    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53ece4b-00fc-4a69-b018-5016eb94676a" path="/var/lib/kubelet/pods/a53ece4b-00fc-4a69-b018-5016eb94676a/volumes"
Oct 31 16:22:56.442071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.442044    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f99a10-a61a-4eac-940f-f2a6f09696fe" path="/var/lib/kubelet/pods/b7f99a10-a61a-4eac-940f-f2a6f09696fe/volumes"
Oct 31 16:22:56.473576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.473551    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"]
Oct 31 16:22:56.474247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.474232    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4fa8ea29-d370-422f-869e-98fb3df77078" containerName="revoke-keys"
Oct 31 16:22:56.474302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.474248    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa8ea29-d370-422f-869e-98fb3df77078" containerName="revoke-keys"
Oct 31 16:22:56.474390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.474380    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa8ea29-d370-422f-869e-98fb3df77078" containerName="revoke-keys"
Oct 31 16:22:56.478415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.478400    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.530588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.529941    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"]
Oct 31 16:22:56.583911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.583883    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.584085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.583950    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.584148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.584079    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngjzn\" (UniqueName: \"kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.685857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.685782    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.686009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.685859    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.686009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.685936    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngjzn\" (UniqueName: \"kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.686722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.686694    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.688350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.688325    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.739583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.739554    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngjzn\" (UniqueName: \"kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn\") pod \"maas-api-revoke-sub-e1641897-sqtb8\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:56.793531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:56.793500    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:57.169294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.169265    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"]
Oct 31 16:22:57.173003 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:57.172974    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf4f5_4d95_4819_bf83_73ab1a700db7.slice/crio-7e4f97441faa605f0a123b37f8d9747cbfe909b85553aae59f9f5ea809de4b37 WatchSource:0}: Error finding container 7e4f97441faa605f0a123b37f8d9747cbfe909b85553aae59f9f5ea809de4b37: Status 404 returned error can't find the container with id 7e4f97441faa605f0a123b37f8d9747cbfe909b85553aae59f9f5ea809de4b37
Oct 31 16:22:57.392383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.392352    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8" event={"ID":"40cdf4f5-4d95-4819-bf83-73ab1a700db7","Type":"ContainerStarted","Data":"7e4f97441faa605f0a123b37f8d9747cbfe909b85553aae59f9f5ea809de4b37"}
Oct 31 16:22:57.546970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.546944    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:57.702585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.702551    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca\") pod \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") "
Oct 31 16:22:57.702685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.702668    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca\") pod \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") "
Oct 31 16:22:57.702925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.702905    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz\" (UniqueName: \"kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz\") pod \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\" (UID: \"07d5cddc-a102-4ba4-a58c-8b1d531865ef\") "
Oct 31 16:22:57.703053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.703030    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca" pod "07d5cddc-a102-4ba4-a58c-8b1d531865ef" (UID: "07d5cddc-a102-4ba4-a58c-8b1d531865ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:57.703740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.703715    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:57.704940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.704910    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca" pod "07d5cddc-a102-4ba4-a58c-8b1d531865ef" (UID: "07d5cddc-a102-4ba4-a58c-8b1d531865ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:57.705117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.705100    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz" pod "07d5cddc-a102-4ba4-a58c-8b1d531865ef" (UID: "07d5cddc-a102-4ba4-a58c-8b1d531865ef"). InnerVolumeSpecName "kube-api-access-h7gpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:57.806932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.806895    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5cddc-a102-4ba4-a58c-8b1d531865ef-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:57.806932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:57.806932    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7gpz\" (UniqueName: \"kubernetes.io/projected/07d5cddc-a102-4ba4-a58c-8b1d531865ef-kube-api-access-h7gpz\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:58.050959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.050924    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"]
Oct 31 16:22:58.055490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.055447    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac69c76-sbr7x"]
Oct 31 16:22:58.401764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.401675    4519 generic.go:356] "Generic (PLEG): container finished" podID="40cdf4f5-4d95-4819-bf83-73ab1a700db7" containerID="b2b02216dc5929bc13a5f369e5265bcda479f802b3317753808197dac7d88fb9" exitCode=0
Oct 31 16:22:58.401913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.401758    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8" event={"ID":"40cdf4f5-4d95-4819-bf83-73ab1a700db7","Type":"ContainerDied","Data":"b2b02216dc5929bc13a5f369e5265bcda479f802b3317753808197dac7d88fb9"}
Oct 31 16:22:58.403181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.403155    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq" event={"ID":"07d5cddc-a102-4ba4-a58c-8b1d531865ef","Type":"ContainerDied","Data":"5ec8b6fe3a817812a0febf188cd171f4834ae4dfd97353b4104feca4baaf283c"}
Oct 31 16:22:58.403334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.403197    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec8b6fe3a817812a0febf188cd171f4834ae4dfd97353b4104feca4baaf283c"
Oct 31 16:22:58.403334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.403199    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"
Oct 31 16:22:58.440381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.440354    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122a1b46-dcb6-40b8-82f7-7554fd55779c" path="/var/lib/kubelet/pods/122a1b46-dcb6-40b8-82f7-7554fd55779c/volumes"
Oct 31 16:22:58.661490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.661391    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"]
Oct 31 16:22:58.662105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.662089    4519 cpu_manager.go:414] "containerMap: removing container" podUID="07d5cddc-a102-4ba4-a58c-8b1d531865ef" containerName="revoke-keys"
Oct 31 16:22:58.662105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.662105    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d5cddc-a102-4ba4-a58c-8b1d531865ef" containerName="revoke-keys"
Oct 31 16:22:58.662236 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.662227    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d5cddc-a102-4ba4-a58c-8b1d531865ef" containerName="revoke-keys"
Oct 31 16:22:58.666108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.666090    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.719186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.719157    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"]
Oct 31 16:22:58.822374 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:22:58.822342    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache]"
Oct 31 16:22:58.823097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.823076    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.823147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.823129    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972jb\" (UniqueName: \"kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.823271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.823249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.925117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.925032    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.925117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.925086    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972jb\" (UniqueName: \"kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.925117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.925112    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.925717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.925694    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.927807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.927777    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.979214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.979183    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972jb\" (UniqueName: \"kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb\") pod \"maas-api-revoke-sub-afa28b1a-bd8ll\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:58.980903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:58.980879    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:22:59.359885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.359856    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"]
Oct 31 16:22:59.363682 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:22:59.363656    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c547c7_c8f7_40b2_9961_e2d14cfbe641.slice/crio-fb074308575ea57e399a17857f6f840d0a5e3ce0c41f5a0bcdb99e765760bca4 WatchSource:0}: Error finding container fb074308575ea57e399a17857f6f840d0a5e3ce0c41f5a0bcdb99e765760bca4: Status 404 returned error can't find the container with id fb074308575ea57e399a17857f6f840d0a5e3ce0c41f5a0bcdb99e765760bca4
Oct 31 16:22:59.412013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.411984    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll" event={"ID":"95c547c7-c8f7-40b2-9961-e2d14cfbe641","Type":"ContainerStarted","Data":"fb074308575ea57e399a17857f6f840d0a5e3ce0c41f5a0bcdb99e765760bca4"}
Oct 31 16:22:59.564477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.564427    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:22:59.634419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.634336    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn\" (UniqueName: \"kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn\") pod \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") "
Oct 31 16:22:59.634593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.634489    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca\") pod \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") "
Oct 31 16:22:59.634703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.634690    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca\") pod \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\" (UID: \"40cdf4f5-4d95-4819-bf83-73ab1a700db7\") "
Oct 31 16:22:59.634842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.634820    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca" pod "40cdf4f5-4d95-4819-bf83-73ab1a700db7" (UID: "40cdf4f5-4d95-4819-bf83-73ab1a700db7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:22:59.635135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.635120    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:59.636523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.636501    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn" pod "40cdf4f5-4d95-4819-bf83-73ab1a700db7" (UID: "40cdf4f5-4d95-4819-bf83-73ab1a700db7"). InnerVolumeSpecName "kube-api-access-ngjzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:22:59.636650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.636630    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca" pod "40cdf4f5-4d95-4819-bf83-73ab1a700db7" (UID: "40cdf4f5-4d95-4819-bf83-73ab1a700db7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:22:59.736984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.736951    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cdf4f5-4d95-4819-bf83-73ab1a700db7-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:22:59.736984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:22:59.736980    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngjzn\" (UniqueName: \"kubernetes.io/projected/40cdf4f5-4d95-4819-bf83-73ab1a700db7-kube-api-access-ngjzn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:00.424072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:00.424037    4519 generic.go:356] "Generic (PLEG): container finished" podID="95c547c7-c8f7-40b2-9961-e2d14cfbe641" containerID="badbeb105513df1aa3e8f5c7a53faf2a3fb31932088e6680455e949af29355f2" exitCode=0
Oct 31 16:23:00.424216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:00.424118    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll" event={"ID":"95c547c7-c8f7-40b2-9961-e2d14cfbe641","Type":"ContainerDied","Data":"badbeb105513df1aa3e8f5c7a53faf2a3fb31932088e6680455e949af29355f2"}
Oct 31 16:23:00.425504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:00.425481    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"
Oct 31 16:23:00.425615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:00.425480    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8" event={"ID":"40cdf4f5-4d95-4819-bf83-73ab1a700db7","Type":"ContainerDied","Data":"7e4f97441faa605f0a123b37f8d9747cbfe909b85553aae59f9f5ea809de4b37"}
Oct 31 16:23:00.425615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:00.425582    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e4f97441faa605f0a123b37f8d9747cbfe909b85553aae59f9f5ea809de4b37"
Oct 31 16:23:01.608696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.608668    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:23:01.767271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.767225    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb\" (UniqueName: \"kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb\") pod \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") "
Oct 31 16:23:01.767525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.767275    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca\") pod \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") "
Oct 31 16:23:01.767525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.767441    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca\") pod \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\" (UID: \"95c547c7-c8f7-40b2-9961-e2d14cfbe641\") "
Oct 31 16:23:01.767862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.767809    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca" pod "95c547c7-c8f7-40b2-9961-e2d14cfbe641" (UID: "95c547c7-c8f7-40b2-9961-e2d14cfbe641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:01.768264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.768239    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:01.769693 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.769666    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb" pod "95c547c7-c8f7-40b2-9961-e2d14cfbe641" (UID: "95c547c7-c8f7-40b2-9961-e2d14cfbe641"). InnerVolumeSpecName "kube-api-access-972jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:01.769852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.769830    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca" pod "95c547c7-c8f7-40b2-9961-e2d14cfbe641" (UID: "95c547c7-c8f7-40b2-9961-e2d14cfbe641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:01.870093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.870060    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972jb\" (UniqueName: \"kubernetes.io/projected/95c547c7-c8f7-40b2-9961-e2d14cfbe641-kube-api-access-972jb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:01.870093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:01.870088    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95c547c7-c8f7-40b2-9961-e2d14cfbe641-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:02.443929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:02.443896    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"
Oct 31 16:23:02.443929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:02.443925    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll" event={"ID":"95c547c7-c8f7-40b2-9961-e2d14cfbe641","Type":"ContainerDied","Data":"fb074308575ea57e399a17857f6f840d0a5e3ce0c41f5a0bcdb99e765760bca4"}
Oct 31 16:23:02.444136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:02.443975    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb074308575ea57e399a17857f6f840d0a5e3ce0c41f5a0bcdb99e765760bca4"
Oct 31 16:23:04.003582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.003551    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"]
Oct 31 16:23:04.004248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.004233    4519 cpu_manager.go:414] "containerMap: removing container" podUID="95c547c7-c8f7-40b2-9961-e2d14cfbe641" containerName="revoke-keys"
Oct 31 16:23:04.004299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.004248    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c547c7-c8f7-40b2-9961-e2d14cfbe641" containerName="revoke-keys"
Oct 31 16:23:04.004299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.004279    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40cdf4f5-4d95-4819-bf83-73ab1a700db7" containerName="revoke-keys"
Oct 31 16:23:04.004299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.004284    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cdf4f5-4d95-4819-bf83-73ab1a700db7" containerName="revoke-keys"
Oct 31 16:23:04.004387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.004378    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cdf4f5-4d95-4819-bf83-73ab1a700db7" containerName="revoke-keys"
Oct 31 16:23:04.004417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.004392    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c547c7-c8f7-40b2-9961-e2d14cfbe641" containerName="revoke-keys"
Oct 31 16:23:04.008239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.008223    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.061492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.061445    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"]
Oct 31 16:23:04.101728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.101697    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:23:04.104514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.104490    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:23:04.104640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.104530    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:23:04.159181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.159150    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:23:04.204281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.204241    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclnt\" (UniqueName: \"kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.204473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.204437    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.204590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.204575    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.306138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.306058    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclnt\" (UniqueName: \"kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.306301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.306183    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.306301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.306277    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.306941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.306915    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.308683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.308647    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.360306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.360278    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclnt\" (UniqueName: \"kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt\") pod \"maas-api-revoke-sub-427c8b07-qlw6b\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:04.621916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:04.621836    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:05.005463 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:05.005428    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"]
Oct 31 16:23:05.008625 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:05.008587    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b99b10c_0d1c_4c52_af9d_c00bb23475ff.slice/crio-902d55041a1882e4c24aa3d2fea7e0f52114bc1e6f2563d1cd3df9b58e2f7dde WatchSource:0}: Error finding container 902d55041a1882e4c24aa3d2fea7e0f52114bc1e6f2563d1cd3df9b58e2f7dde: Status 404 returned error can't find the container with id 902d55041a1882e4c24aa3d2fea7e0f52114bc1e6f2563d1cd3df9b58e2f7dde
Oct 31 16:23:05.009963 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:05.009938    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:05.484710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:05.484672    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b" event={"ID":"8b99b10c-0d1c-4c52-af9d-c00bb23475ff","Type":"ContainerStarted","Data":"b8ca320daf40f9b51ce9f938a05b1c0491c11af42166790af1a53130b9bd4b9b"}
Oct 31 16:23:05.484862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:05.484731    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b" event={"ID":"8b99b10c-0d1c-4c52-af9d-c00bb23475ff","Type":"ContainerStarted","Data":"902d55041a1882e4c24aa3d2fea7e0f52114bc1e6f2563d1cd3df9b58e2f7dde"}
Oct 31 16:23:05.593940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:05.593845    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b" podStartSLOduration=2.593829598 podStartE2EDuration="2.593829598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:05.592672337 +0000 UTC m=+1229.292601648" watchObservedRunningTime="2026-10-31 16:23:05.593829598 +0000 UTC m=+1229.293758908"
Oct 31 16:23:06.494580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:06.494549    4519 generic.go:356] "Generic (PLEG): container finished" podID="8b99b10c-0d1c-4c52-af9d-c00bb23475ff" containerID="b8ca320daf40f9b51ce9f938a05b1c0491c11af42166790af1a53130b9bd4b9b" exitCode=0
Oct 31 16:23:06.494966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:06.494625    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b" event={"ID":"8b99b10c-0d1c-4c52-af9d-c00bb23475ff","Type":"ContainerDied","Data":"b8ca320daf40f9b51ce9f938a05b1c0491c11af42166790af1a53130b9bd4b9b"}
Oct 31 16:23:07.074974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.074941    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"]
Oct 31 16:23:07.079491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.079443    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.135329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.135295    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"]
Oct 31 16:23:07.148317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148286    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.148515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148417    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.148583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148547    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzv6\" (UniqueName: \"kubernetes.io/projected/c269346d-fb60-416d-9f40-f3ef2530a5d5-kube-api-access-9tzv6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.148660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148633    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.148757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148738    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.148801 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148783    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c269346d-fb60-416d-9f40-f3ef2530a5d5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.148946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.148918    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.174703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.174666    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 16:23:07.174703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.174666    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 16:23:07.174703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.174702    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 16:23:07.174912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.174715    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2bg6m\""
Oct 31 16:23:07.250343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250312    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.250532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250384    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.250532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250483    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzv6\" (UniqueName: \"kubernetes.io/projected/c269346d-fb60-416d-9f40-f3ef2530a5d5-kube-api-access-9tzv6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.250688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250664    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.250810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250784    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.250965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250839    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c269346d-fb60-416d-9f40-f3ef2530a5d5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.251036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.250987    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.251930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.251902    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.253044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.253013    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.253143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.253079    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.253143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.253096    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c269346d-fb60-416d-9f40-f3ef2530a5d5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.253263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.253216    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.310433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.310393    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c269346d-fb60-416d-9f40-f3ef2530a5d5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.311706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.311679    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzv6\" (UniqueName: \"kubernetes.io/projected/c269346d-fb60-416d-9f40-f3ef2530a5d5-kube-api-access-9tzv6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9d8tx\" (UID: \"c269346d-fb60-416d-9f40-f3ef2530a5d5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.393654 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.393576    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:07.660578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.660552    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:07.759331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.759295    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca\") pod \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") "
Oct 31 16:23:07.759589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.759340    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt\" (UniqueName: \"kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt\") pod \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") "
Oct 31 16:23:07.759635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.759614    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca\") pod \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\" (UID: \"8b99b10c-0d1c-4c52-af9d-c00bb23475ff\") "
Oct 31 16:23:07.759979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.759953    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca" pod "8b99b10c-0d1c-4c52-af9d-c00bb23475ff" (UID: "8b99b10c-0d1c-4c52-af9d-c00bb23475ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:07.760169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.760148    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:07.761449 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.761416    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca" pod "8b99b10c-0d1c-4c52-af9d-c00bb23475ff" (UID: "8b99b10c-0d1c-4c52-af9d-c00bb23475ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:07.761584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.761528    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt" pod "8b99b10c-0d1c-4c52-af9d-c00bb23475ff" (UID: "8b99b10c-0d1c-4c52-af9d-c00bb23475ff"). InnerVolumeSpecName "kube-api-access-hclnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:07.830520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.830482    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"]
Oct 31 16:23:07.833327 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:07.833301    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc269346d_fb60_416d_9f40_f3ef2530a5d5.slice/crio-e9526b2cab46d15847b8828a4a0ba07cb192176517262cd4466baa7234f4c546 WatchSource:0}: Error finding container e9526b2cab46d15847b8828a4a0ba07cb192176517262cd4466baa7234f4c546: Status 404 returned error can't find the container with id e9526b2cab46d15847b8828a4a0ba07cb192176517262cd4466baa7234f4c546
Oct 31 16:23:07.835628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.835609    4519 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:23:07.862116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.862087    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:07.862116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:07.862115    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hclnt\" (UniqueName: \"kubernetes.io/projected/8b99b10c-0d1c-4c52-af9d-c00bb23475ff-kube-api-access-hclnt\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:08.053773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.053725    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"]
Oct 31 16:23:08.058656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.058635    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bedb1-56ch6"]
Oct 31 16:23:08.063084 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.063059    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"]
Oct 31 16:23:08.083246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.083211    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd41ca27-tnqzk"]
Oct 31 16:23:08.440407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.440332    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60877a1b-0839-4e9b-8c13-348fc4f46f25" path="/var/lib/kubelet/pods/60877a1b-0839-4e9b-8c13-348fc4f46f25/volumes"
Oct 31 16:23:08.440854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.440839    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17c3e5a-843d-4c85-b16f-13701843e1c9" path="/var/lib/kubelet/pods/e17c3e5a-843d-4c85-b16f-13701843e1c9/volumes"
Oct 31 16:23:08.525218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.525182    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx" event={"ID":"c269346d-fb60-416d-9f40-f3ef2530a5d5","Type":"ContainerStarted","Data":"e9526b2cab46d15847b8828a4a0ba07cb192176517262cd4466baa7234f4c546"}
Oct 31 16:23:08.527074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.527042    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b" event={"ID":"8b99b10c-0d1c-4c52-af9d-c00bb23475ff","Type":"ContainerDied","Data":"902d55041a1882e4c24aa3d2fea7e0f52114bc1e6f2563d1cd3df9b58e2f7dde"}
Oct 31 16:23:08.527225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.527095    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902d55041a1882e4c24aa3d2fea7e0f52114bc1e6f2563d1cd3df9b58e2f7dde"
Oct 31 16:23:08.527225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:08.527127    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"
Oct 31 16:23:09.322324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.322291    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"]
Oct 31 16:23:09.323205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.323147    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8b99b10c-0d1c-4c52-af9d-c00bb23475ff" containerName="revoke-keys"
Oct 31 16:23:09.323205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.323169    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b99b10c-0d1c-4c52-af9d-c00bb23475ff" containerName="revoke-keys"
Oct 31 16:23:09.323365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.323358    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b99b10c-0d1c-4c52-af9d-c00bb23475ff" containerName="revoke-keys"
Oct 31 16:23:09.328710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.328691    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.381041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.380594    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"]
Oct 31 16:23:09.385511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.385440    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtlv\" (UniqueName: \"kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.385657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.385631    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.385722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.385671    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.421629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.421491    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:23:09.421629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.421542    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:23:09.422091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.422072    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:23:09.425209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.425188    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:23:09.487621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.487590    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.487791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.487638    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.487990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.487968    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtlv\" (UniqueName: \"kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.488326 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.488305    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.489982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.489959    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.542779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.542752    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtlv\" (UniqueName: \"kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv\") pod \"maas-api-revoke-sub-6f3d3f21-js5hk\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:09.645359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:09.645280    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:10.026640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:10.026527    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"]
Oct 31 16:23:10.030169 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:10.030137    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28681382_119d_4b1c_ba27_bb0196abf0ac.slice/crio-71aa39ab6059c003b9b33945bdcd54619dc517fb39328c5c44bc21b2a8dc1c12 WatchSource:0}: Error finding container 71aa39ab6059c003b9b33945bdcd54619dc517fb39328c5c44bc21b2a8dc1c12: Status 404 returned error can't find the container with id 71aa39ab6059c003b9b33945bdcd54619dc517fb39328c5c44bc21b2a8dc1c12
Oct 31 16:23:10.549756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:10.549653    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk" event={"ID":"28681382-119d-4b1c-ba27-bb0196abf0ac","Type":"ContainerStarted","Data":"35df525b64813c08a73701dced17cf1cc49a953afafc96d953a17adc3668278c"}
Oct 31 16:23:10.549756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:10.549733    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk" event={"ID":"28681382-119d-4b1c-ba27-bb0196abf0ac","Type":"ContainerStarted","Data":"71aa39ab6059c003b9b33945bdcd54619dc517fb39328c5c44bc21b2a8dc1c12"}
Oct 31 16:23:10.704743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:10.704681    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk" podStartSLOduration=1.704663217 podStartE2EDuration="1.704663217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:10.656704756 +0000 UTC m=+1234.356634067" watchObservedRunningTime="2026-10-31 16:23:10.704663217 +0000 UTC m=+1234.404592529"
Oct 31 16:23:11.562113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:11.562078    4519 generic.go:356] "Generic (PLEG): container finished" podID="28681382-119d-4b1c-ba27-bb0196abf0ac" containerID="35df525b64813c08a73701dced17cf1cc49a953afafc96d953a17adc3668278c" exitCode=0
Oct 31 16:23:11.562542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:11.562167    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk" event={"ID":"28681382-119d-4b1c-ba27-bb0196abf0ac","Type":"ContainerDied","Data":"35df525b64813c08a73701dced17cf1cc49a953afafc96d953a17adc3668278c"}
Oct 31 16:23:12.047278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.047241    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"]
Oct 31 16:23:12.052833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.052799    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e594afa1-9v7v4"]
Oct 31 16:23:12.229252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.229216    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:12.229412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.229300    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:12.440996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.440965    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e0e1db-8ebe-40e6-8188-63f1498e49a4" path="/var/lib/kubelet/pods/b4e0e1db-8ebe-40e6-8188-63f1498e49a4/volumes"
Oct 31 16:23:12.573039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.572939    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx" event={"ID":"c269346d-fb60-416d-9f40-f3ef2530a5d5","Type":"ContainerStarted","Data":"07ff70d1c3904705d0466f833554d5a79783acc74d01eb0b956cb25ffa1ab17e"}
Oct 31 16:23:12.573519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.573162    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:12.646448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.646416    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"]
Oct 31 16:23:12.651495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.651469    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.689103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.689043    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx" podStartSLOduration=1.2969572120000001 podStartE2EDuration="5.689028905s" totalImagesPullingTime="4.392071693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:12.682876453 +0000 UTC m=+1236.382805782" watchObservedRunningTime="2026-10-31 16:23:12.689028905 +0000 UTC m=+1236.388958216"
Oct 31 16:23:12.706627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.706596    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"]
Oct 31 16:23:12.745592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.745559    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.745942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.745699    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.745942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.745780    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwllg\" (UniqueName: \"kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.756253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.756229    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:12.847020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.846937    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv\" (UniqueName: \"kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv\") pod \"28681382-119d-4b1c-ba27-bb0196abf0ac\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") "
Oct 31 16:23:12.847196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.847140    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca\") pod \"28681382-119d-4b1c-ba27-bb0196abf0ac\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") "
Oct 31 16:23:12.847196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.847177    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca\") pod \"28681382-119d-4b1c-ba27-bb0196abf0ac\" (UID: \"28681382-119d-4b1c-ba27-bb0196abf0ac\") "
Oct 31 16:23:12.847608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.847567    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca" pod "28681382-119d-4b1c-ba27-bb0196abf0ac" (UID: "28681382-119d-4b1c-ba27-bb0196abf0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:12.849015 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.848985    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.849340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.849306    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.849564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.849545    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwllg\" (UniqueName: \"kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.849899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.849880    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:12.850183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.850161    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv" pod "28681382-119d-4b1c-ba27-bb0196abf0ac" (UID: "28681382-119d-4b1c-ba27-bb0196abf0ac"). InnerVolumeSpecName "kube-api-access-nrtlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:12.850843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.850816    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.851799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.851766    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca" pod "28681382-119d-4b1c-ba27-bb0196abf0ac" (UID: "28681382-119d-4b1c-ba27-bb0196abf0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:12.853260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.853239    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.904850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.904816    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwllg\" (UniqueName: \"kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg\") pod \"maas-api-revoke-sub-55df8775-48ldc\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:12.951867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.951819    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtlv\" (UniqueName: \"kubernetes.io/projected/28681382-119d-4b1c-ba27-bb0196abf0ac-kube-api-access-nrtlv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:12.951867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.951862    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28681382-119d-4b1c-ba27-bb0196abf0ac-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:12.983576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:12.983550    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:13.366694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.366659    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"]
Oct 31 16:23:13.370216 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:13.370180    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27a37e3_fd04_4e4b_acbf_297df257d4c9.slice/crio-489759cbbab6091bae7ff6ced0a48ca51719eb24509b63ef9909594433da1b88 WatchSource:0}: Error finding container 489759cbbab6091bae7ff6ced0a48ca51719eb24509b63ef9909594433da1b88: Status 404 returned error can't find the container with id 489759cbbab6091bae7ff6ced0a48ca51719eb24509b63ef9909594433da1b88
Oct 31 16:23:13.586428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.586390    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc" event={"ID":"b27a37e3-fd04-4e4b-acbf-297df257d4c9","Type":"ContainerStarted","Data":"489759cbbab6091bae7ff6ced0a48ca51719eb24509b63ef9909594433da1b88"}
Oct 31 16:23:13.588600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.588577    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"
Oct 31 16:23:13.588717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.588600    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk" event={"ID":"28681382-119d-4b1c-ba27-bb0196abf0ac","Type":"ContainerDied","Data":"71aa39ab6059c003b9b33945bdcd54619dc517fb39328c5c44bc21b2a8dc1c12"}
Oct 31 16:23:13.588717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.588651    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71aa39ab6059c003b9b33945bdcd54619dc517fb39328c5c44bc21b2a8dc1c12"
Oct 31 16:23:13.590862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.590837    4519 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9d8tx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:23:13.590975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:13.590883    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx" podUID="c269346d-fb60-416d-9f40-f3ef2530a5d5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:23:13.922806 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:13.922768    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:14.598982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:14.598947    4519 generic.go:356] "Generic (PLEG): container finished" podID="b27a37e3-fd04-4e4b-acbf-297df257d4c9" containerID="7c08785f67ce3c509b120ce72c8818425b74a9de701c01180601dc7c96472c25" exitCode=0
Oct 31 16:23:14.599370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:14.599037    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc" event={"ID":"b27a37e3-fd04-4e4b-acbf-297df257d4c9","Type":"ContainerDied","Data":"7c08785f67ce3c509b120ce72c8818425b74a9de701c01180601dc7c96472c25"}
Oct 31 16:23:15.026467 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:15.026407    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:15.045326 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:15.045287    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"]
Oct 31 16:23:15.049973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:15.049946    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b54f894-rrljb"]
Oct 31 16:23:15.852772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:15.847831    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:16.071753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.064051    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca\") pod \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") "
Oct 31 16:23:16.071753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.064272    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca\") pod \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") "
Oct 31 16:23:16.071753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.064370    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg\" (UniqueName: \"kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg\") pod \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\" (UID: \"b27a37e3-fd04-4e4b-acbf-297df257d4c9\") "
Oct 31 16:23:16.076874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.076050    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg" pod "b27a37e3-fd04-4e4b-acbf-297df257d4c9" (UID: "b27a37e3-fd04-4e4b-acbf-297df257d4c9"). InnerVolumeSpecName "kube-api-access-vwllg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:16.076874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.076431    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca" pod "b27a37e3-fd04-4e4b-acbf-297df257d4c9" (UID: "b27a37e3-fd04-4e4b-acbf-297df257d4c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:16.129684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.129643    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca" pod "b27a37e3-fd04-4e4b-acbf-297df257d4c9" (UID: "b27a37e3-fd04-4e4b-acbf-297df257d4c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:16.211510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.208425    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:16.211510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.208491    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwllg\" (UniqueName: \"kubernetes.io/projected/b27a37e3-fd04-4e4b-acbf-297df257d4c9-kube-api-access-vwllg\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:16.211510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.208510    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27a37e3-fd04-4e4b-acbf-297df257d4c9-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.252368    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"]
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.253529    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b27a37e3-fd04-4e4b-acbf-297df257d4c9" containerName="revoke-keys"
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.253550    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27a37e3-fd04-4e4b-acbf-297df257d4c9" containerName="revoke-keys"
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.253584    4519 cpu_manager.go:414] "containerMap: removing container" podUID="28681382-119d-4b1c-ba27-bb0196abf0ac" containerName="revoke-keys"
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.253592    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28681382-119d-4b1c-ba27-bb0196abf0ac" containerName="revoke-keys"
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.253890    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27a37e3-fd04-4e4b-acbf-297df257d4c9" containerName="revoke-keys"
Oct 31 16:23:16.268482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.253908    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="28681382-119d-4b1c-ba27-bb0196abf0ac" containerName="revoke-keys"
Oct 31 16:23:16.592479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.591983    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:16.693490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.691572    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lbl\" (UniqueName: \"kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:16.693490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.691644    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:16.693490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.691768    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:16.841479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.838965    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"]
Oct 31 16:23:17.020480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.998953    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lbl\" (UniqueName: \"kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.020480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.999045    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.020480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:16.999245    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.055092 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.023375    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.096481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.096109    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"
Oct 31 16:23:17.251512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.244501    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.289557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.271075    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lbl\" (UniqueName: \"kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl\") pod \"maas-api-revoke-sub-9148f6d7-46hjk\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.370327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.368560    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.370327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.369701    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.371410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.371384    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08c7e70-d9d7-4283-8194-8b855d3030ae" path="/var/lib/kubelet/pods/c08c7e70-d9d7-4283-8194-8b855d3030ae/volumes"
Oct 31 16:23:17.399274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.399043    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc" event={"ID":"b27a37e3-fd04-4e4b-acbf-297df257d4c9","Type":"ContainerDied","Data":"489759cbbab6091bae7ff6ced0a48ca51719eb24509b63ef9909594433da1b88"}
Oct 31 16:23:17.399274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.399118    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489759cbbab6091bae7ff6ced0a48ca51719eb24509b63ef9909594433da1b88"
Oct 31 16:23:17.399274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.399189    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9d8tx"
Oct 31 16:23:17.435540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.410773    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.435540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.425753    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:17.478533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.475358    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.478862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.478829    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.574031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.554793    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.574031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.555167    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.621594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.621543    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.622188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.622156    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.622616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.622592    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.914162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:17.914121    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.007030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.006563    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.417489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.386565    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.417489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.388429    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:23:18.417489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.407716    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.417489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.408096    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:23:18.418157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.417527    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.489567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.419371    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:23:18.489567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.421327    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.533486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.491308    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.536258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.533763    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.573612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.573572    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.663354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.602367    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.769633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.759608    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.820681 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.797654    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.843940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.820608    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.853498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.797793    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.964479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.917701    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.047534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:18.975521    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.047534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.007633    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:23:19.429875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.428538    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:23:19.429875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.429256    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.460527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.455517    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.484652 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:19.484605    4519 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.052s"
Oct 31 16:23:19.485020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.484991    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:23:19.485225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.485210    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:23:19.485407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.485385    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:23:19.485582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.485561    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:23:19.488400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.488381    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:23:19.510479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.496266    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.522491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.496956    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:23:19.522798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.522776    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:23:19.531597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.531567    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:23:19.556597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.556563    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:23:19.598481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.541330    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.599105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.588013    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.739930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.634589    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.890134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.873530    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.890134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.886919    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:23:19.890134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.887261    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.952880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:19.952099    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.011695 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.011592    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.012292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.012257    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.444516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.430149    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.444516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.430628    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.477492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.464775    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.613886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.531858    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.698502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.694309    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.733652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.733603    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.777285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.753545    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.921555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.899818    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.921555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.900131    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.963650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:20.963606    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.017606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.017559    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.048077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.048030    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.049597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.049569    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:21.049771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.049747    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.412485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.397136    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"]
Oct 31 16:23:21.445618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.445562    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.908884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.466568    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.908884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.466630    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.910714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.909251    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.989849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.986602    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.013610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:21.995891    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.023112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.018528    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.047389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.024566    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.170056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.157747    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.240183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.240146    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk" event={"ID":"c70cc646-f815-4957-a15a-2e6f8aa34529","Type":"ContainerStarted","Data":"2cd701de5666ea2c9f8f87a390b2b9053abcda8eb31841b2aa183f4e6e969db2"}
Oct 31 16:23:22.263045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.262992    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 16:23:22.263599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.263568    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.264404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.264372    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.441534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.159437    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.877920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.877889    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:23:22.880784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.880702    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:23:22.882155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.882133    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:23:22.882345 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.882330    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:23:22.882504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.882492    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:23:22.882638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.882628    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:23:22.883912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.883894    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:23:22.884090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.884078    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:23:22.887065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.887048    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:23:22.887222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.887210    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:23:22.887591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.881732    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.916548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.916521    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"]
Oct 31 16:23:22.921468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.920743    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00359b39-sgz7m"]
Oct 31 16:23:22.925315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.925294    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:23:22.925532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:22.925517    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:23:23.157601 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:23.157565    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:23.157778 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:23.157578    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:23.254942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.254907    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk" event={"ID":"c70cc646-f815-4957-a15a-2e6f8aa34529","Type":"ContainerStarted","Data":"c80ed7904352ff6cb25410d5799b8dfd162a28f97019e40b49198873068de289"}
Oct 31 16:23:23.452092 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.451148    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"]
Oct 31 16:23:23.456989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.456965    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.508097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.508069    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"]
Oct 31 16:23:23.543593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.543554    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.543859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.543836    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.544343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.544311    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.646101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.646068    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.646287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.646190    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.646369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.646353    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.647020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.646994    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.648767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.648745    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.699698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.699668    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj\") pod \"maas-api-revoke-sub-fc938915-k67gq\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:23.820758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:23.820728    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:24.013740 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:24.013698    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:23:24.220042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.220013    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"]
Oct 31 16:23:24.222977 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:24.222954    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5547d5cc_7dbc_464e_9fcd_a3e23bb3c46e.slice/crio-9ed420df915029be698426b2f555a508a185718e9d16d225e1f81c744bd5395c WatchSource:0}: Error finding container 9ed420df915029be698426b2f555a508a185718e9d16d225e1f81c744bd5395c: Status 404 returned error can't find the container with id 9ed420df915029be698426b2f555a508a185718e9d16d225e1f81c744bd5395c
Oct 31 16:23:24.266652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.266622    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq" event={"ID":"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e","Type":"ContainerStarted","Data":"9ed420df915029be698426b2f555a508a185718e9d16d225e1f81c744bd5395c"}
Oct 31 16:23:24.268354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.268324    4519 generic.go:356] "Generic (PLEG): container finished" podID="c70cc646-f815-4957-a15a-2e6f8aa34529" containerID="c80ed7904352ff6cb25410d5799b8dfd162a28f97019e40b49198873068de289" exitCode=0
Oct 31 16:23:24.268470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.268403    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk" event={"ID":"c70cc646-f815-4957-a15a-2e6f8aa34529","Type":"ContainerDied","Data":"c80ed7904352ff6cb25410d5799b8dfd162a28f97019e40b49198873068de289"}
Oct 31 16:23:24.287868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.287825    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk" podStartSLOduration=8.287812582 podStartE2EDuration="8.287812582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:24.284601011 +0000 UTC m=+1247.984530323" watchObservedRunningTime="2026-10-31 16:23:24.287812582 +0000 UTC m=+1247.987741894"
Oct 31 16:23:24.371208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.371176    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"]
Oct 31 16:23:24.377135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.377113    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.429669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.429625    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"]
Oct 31 16:23:24.451502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.451446    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373bd2d7-edc3-4bac-9d65-53d901f46d09" path="/var/lib/kubelet/pods/373bd2d7-edc3-4bac-9d65-53d901f46d09/volumes"
Oct 31 16:23:24.461065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.461026    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.461243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.461108    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.461312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.461283    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8wq\" (UniqueName: \"kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.563442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.563369    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8wq\" (UniqueName: \"kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.563660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.563643    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.563740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.563691    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.564403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.564379    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.566319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.566293    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.617687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.617659    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8wq\" (UniqueName: \"kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq\") pod \"maas-api-revoke-sub-6a210d9b-rkghn\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:24.700431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:24.700399    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:25.045145 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:25.045109    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:25.080175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.080148    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"]
Oct 31 16:23:25.082543 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:25.082516    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6fc860_d3a2_489b_b1d6_0bf22aa3ffa8.slice/crio-ac11b377f1a7d1713533a5972ee2dea30e5645218542d0b1c977d22315307f28 WatchSource:0}: Error finding container ac11b377f1a7d1713533a5972ee2dea30e5645218542d0b1c977d22315307f28: Status 404 returned error can't find the container with id ac11b377f1a7d1713533a5972ee2dea30e5645218542d0b1c977d22315307f28
Oct 31 16:23:25.291821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.291782    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn" event={"ID":"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8","Type":"ContainerStarted","Data":"ac11b377f1a7d1713533a5972ee2dea30e5645218542d0b1c977d22315307f28"}
Oct 31 16:23:25.293615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.293587    4519 generic.go:356] "Generic (PLEG): container finished" podID="5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" containerID="94a4fc2ea983d5c7ef7e7ee6861f0b91c991859254657d1615414dc753617f99" exitCode=0
Oct 31 16:23:25.293752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.293664    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq" event={"ID":"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e","Type":"ContainerDied","Data":"94a4fc2ea983d5c7ef7e7ee6861f0b91c991859254657d1615414dc753617f99"}
Oct 31 16:23:25.444451 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.444423    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:25.581148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.581118    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca\") pod \"c70cc646-f815-4957-a15a-2e6f8aa34529\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") "
Oct 31 16:23:25.581424 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.581409    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl\" (UniqueName: \"kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl\") pod \"c70cc646-f815-4957-a15a-2e6f8aa34529\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") "
Oct 31 16:23:25.581871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.581491    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca\") pod \"c70cc646-f815-4957-a15a-2e6f8aa34529\" (UID: \"c70cc646-f815-4957-a15a-2e6f8aa34529\") "
Oct 31 16:23:25.581945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.581912    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca" pod "c70cc646-f815-4957-a15a-2e6f8aa34529" (UID: "c70cc646-f815-4957-a15a-2e6f8aa34529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:25.582119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.582095    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:25.583624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.583600    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl" pod "c70cc646-f815-4957-a15a-2e6f8aa34529" (UID: "c70cc646-f815-4957-a15a-2e6f8aa34529"). InnerVolumeSpecName "kube-api-access-f8lbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:25.583792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.583769    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca" pod "c70cc646-f815-4957-a15a-2e6f8aa34529" (UID: "c70cc646-f815-4957-a15a-2e6f8aa34529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:25.684115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.684044    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lbl\" (UniqueName: \"kubernetes.io/projected/c70cc646-f815-4957-a15a-2e6f8aa34529-kube-api-access-f8lbl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:25.684115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:25.684071    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70cc646-f815-4957-a15a-2e6f8aa34529-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:26.234505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.234449    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"]
Oct 31 16:23:26.235202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.235186    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c70cc646-f815-4957-a15a-2e6f8aa34529" containerName="revoke-keys"
Oct 31 16:23:26.235246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.235203    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70cc646-f815-4957-a15a-2e6f8aa34529" containerName="revoke-keys"
Oct 31 16:23:26.235362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.235353    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70cc646-f815-4957-a15a-2e6f8aa34529" containerName="revoke-keys"
Oct 31 16:23:26.239788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.239770    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.291576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.291543    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"]
Oct 31 16:23:26.310479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.307006    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"
Oct 31 16:23:26.310479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.307018    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk" event={"ID":"c70cc646-f815-4957-a15a-2e6f8aa34529","Type":"ContainerDied","Data":"2cd701de5666ea2c9f8f87a390b2b9053abcda8eb31841b2aa183f4e6e969db2"}
Oct 31 16:23:26.310479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.307436    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cd701de5666ea2c9f8f87a390b2b9053abcda8eb31841b2aa183f4e6e969db2"
Oct 31 16:23:26.310479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.309585    4519 generic.go:356] "Generic (PLEG): container finished" podID="5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" containerID="5f8136305e890d99394da236f4657826532e2af68eb2fefdde0426eca8a132b2" exitCode=0
Oct 31 16:23:26.310479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.309668    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn" event={"ID":"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8","Type":"ContainerDied","Data":"5f8136305e890d99394da236f4657826532e2af68eb2fefdde0426eca8a132b2"}
Oct 31 16:23:26.394745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.394702    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66x96\" (UniqueName: \"kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.395113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.395090    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.395244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.395166    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.496492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.496424    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.496627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.496521    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.496627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.496563    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66x96\" (UniqueName: \"kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.497178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.497151    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.498828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.498805    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.498946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.498905    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:26.550551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.550522    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66x96\" (UniqueName: \"kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96\") pod \"maas-api-revoke-sub-677da01e-xshpn\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.555361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.555336    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:26.598055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.598017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca\") pod \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") "
Oct 31 16:23:26.598277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.598259    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj\") pod \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") "
Oct 31 16:23:26.598353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.598300    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca\") pod \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\" (UID: \"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e\") "
Oct 31 16:23:26.598409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.598355    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca" pod "5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" (UID: "5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:26.598996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.598976    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:26.600736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.600696    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj" pod "5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" (UID: "5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e"). InnerVolumeSpecName "kube-api-access-tlfjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:26.604581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.604559    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca" pod "5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" (UID: "5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:26.700299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.700210    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:26.700299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.700241    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e-kube-api-access-tlfjj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:26.933780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:26.933754    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"]
Oct 31 16:23:26.936732 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:26.936710    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25223c0f_3844_4822_b0cf_11ba4beff2cd.slice/crio-3ac5953f76fe4e65d7a9f26b394db9f552fbabababffd6024411220b7f0f60dc WatchSource:0}: Error finding container 3ac5953f76fe4e65d7a9f26b394db9f552fbabababffd6024411220b7f0f60dc: Status 404 returned error can't find the container with id 3ac5953f76fe4e65d7a9f26b394db9f552fbabababffd6024411220b7f0f60dc
Oct 31 16:23:27.328508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.328430    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn" event={"ID":"25223c0f-3844-4822-b0cf-11ba4beff2cd","Type":"ContainerStarted","Data":"94d6e92f1115a77682d5d944a888605fc0dbc1804f57ba241e14f5f0c1650970"}
Oct 31 16:23:27.328981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.328556    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn" event={"ID":"25223c0f-3844-4822-b0cf-11ba4beff2cd","Type":"ContainerStarted","Data":"3ac5953f76fe4e65d7a9f26b394db9f552fbabababffd6024411220b7f0f60dc"}
Oct 31 16:23:27.329971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.329950    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"
Oct 31 16:23:27.330088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.329972    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq" event={"ID":"5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e","Type":"ContainerDied","Data":"9ed420df915029be698426b2f555a508a185718e9d16d225e1f81c744bd5395c"}
Oct 31 16:23:27.330088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.330016    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed420df915029be698426b2f555a508a185718e9d16d225e1f81c744bd5395c"
Oct 31 16:23:27.502646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.502624    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:27.609873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.609776    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn" podStartSLOduration=1.609762045 podStartE2EDuration="1.609762045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:27.442574815 +0000 UTC m=+1251.142504129" watchObservedRunningTime="2026-10-31 16:23:27.609762045 +0000 UTC m=+1251.309691356"
Oct 31 16:23:27.615946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.615920    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca\") pod \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") "
Oct 31 16:23:27.616109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.616014    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq\" (UniqueName: \"kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq\") pod \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") "
Oct 31 16:23:27.616109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.616058    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca\") pod \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\" (UID: \"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8\") "
Oct 31 16:23:27.619013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.616416    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca" pod "5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" (UID: "5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:27.619013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.616770    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:27.619013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.618361    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca" pod "5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" (UID: "5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:27.641747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.641714    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq" pod "5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" (UID: "5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8"). InnerVolumeSpecName "kube-api-access-gj8wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:27.718562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.718529    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:27.718562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:27.718559    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8wq\" (UniqueName: \"kubernetes.io/projected/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8-kube-api-access-gj8wq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:28.340066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.340039    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"
Oct 31 16:23:28.340901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.340088    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn" event={"ID":"5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8","Type":"ContainerDied","Data":"ac11b377f1a7d1713533a5972ee2dea30e5645218542d0b1c977d22315307f28"}
Oct 31 16:23:28.340901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.340137    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac11b377f1a7d1713533a5972ee2dea30e5645218542d0b1c977d22315307f28"
Oct 31 16:23:28.342148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.342123    4519 generic.go:356] "Generic (PLEG): container finished" podID="25223c0f-3844-4822-b0cf-11ba4beff2cd" containerID="94d6e92f1115a77682d5d944a888605fc0dbc1804f57ba241e14f5f0c1650970" exitCode=0
Oct 31 16:23:28.342289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.342153    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn" event={"ID":"25223c0f-3844-4822-b0cf-11ba4beff2cd","Type":"ContainerDied","Data":"94d6e92f1115a77682d5d944a888605fc0dbc1804f57ba241e14f5f0c1650970"}
Oct 31 16:23:28.552374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.552342    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"]
Oct 31 16:23:28.553146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.553124    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" containerName="revoke-keys"
Oct 31 16:23:28.553146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.553147    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" containerName="revoke-keys"
Oct 31 16:23:28.553311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.553236    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" containerName="revoke-keys"
Oct 31 16:23:28.553311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.553244    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" containerName="revoke-keys"
Oct 31 16:23:28.553442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.553426    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" containerName="revoke-keys"
Oct 31 16:23:28.553523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.553447    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" containerName="revoke-keys"
Oct 31 16:23:28.556909 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.556888    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.609331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.609266    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"]
Oct 31 16:23:28.733315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.733279    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtjv\" (UniqueName: \"kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.733548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.733465    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.733690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.733672    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.834575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.834544    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.834733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.834686    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.834821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.834804    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtjv\" (UniqueName: \"kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.835401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.835375    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.837000 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.836981    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.897724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:28.897654    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtjv\" (UniqueName: \"kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv\") pod \"maas-api-revoke-sub-e79ec77a-wq76n\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:28.897856 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:28.897830    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:29.171494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.171399    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:29.528609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.528585    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:29.554968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.554940    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"]
Oct 31 16:23:29.557757 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:29.557733    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce328c5d_1aba_4298_99ef_8a06752141e6.slice/crio-9b7dea450f878c2ec201ad4aceefcc7cce89f41407c113be16222d6069feeff6 WatchSource:0}: Error finding container 9b7dea450f878c2ec201ad4aceefcc7cce89f41407c113be16222d6069feeff6: Status 404 returned error can't find the container with id 9b7dea450f878c2ec201ad4aceefcc7cce89f41407c113be16222d6069feeff6
Oct 31 16:23:29.646649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.646617    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca\") pod \"25223c0f-3844-4822-b0cf-11ba4beff2cd\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") "
Oct 31 16:23:29.646786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.646683    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96\" (UniqueName: \"kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96\") pod \"25223c0f-3844-4822-b0cf-11ba4beff2cd\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") "
Oct 31 16:23:29.646786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.646733    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca\") pod \"25223c0f-3844-4822-b0cf-11ba4beff2cd\" (UID: \"25223c0f-3844-4822-b0cf-11ba4beff2cd\") "
Oct 31 16:23:29.647122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.647096    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca" pod "25223c0f-3844-4822-b0cf-11ba4beff2cd" (UID: "25223c0f-3844-4822-b0cf-11ba4beff2cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:29.647377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.647361    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:29.648651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.648624    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca" pod "25223c0f-3844-4822-b0cf-11ba4beff2cd" (UID: "25223c0f-3844-4822-b0cf-11ba4beff2cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:29.648862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.648845    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96" pod "25223c0f-3844-4822-b0cf-11ba4beff2cd" (UID: "25223c0f-3844-4822-b0cf-11ba4beff2cd"). InnerVolumeSpecName "kube-api-access-66x96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:29.748371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.748341    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25223c0f-3844-4822-b0cf-11ba4beff2cd-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:29.748371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:29.748369    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66x96\" (UniqueName: \"kubernetes.io/projected/25223c0f-3844-4822-b0cf-11ba4beff2cd-kube-api-access-66x96\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:30.045483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.045394    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"]
Oct 31 16:23:30.050080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.050055    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a52cba53-444p8"]
Oct 31 16:23:30.360810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.360733    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"
Oct 31 16:23:30.360810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.360763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn" event={"ID":"25223c0f-3844-4822-b0cf-11ba4beff2cd","Type":"ContainerDied","Data":"3ac5953f76fe4e65d7a9f26b394db9f552fbabababffd6024411220b7f0f60dc"}
Oct 31 16:23:30.361047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.360816    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac5953f76fe4e65d7a9f26b394db9f552fbabababffd6024411220b7f0f60dc"
Oct 31 16:23:30.362478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.362437    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n" event={"ID":"ce328c5d-1aba-4298-99ef-8a06752141e6","Type":"ContainerStarted","Data":"adc387ad69f85c4ce64edc08a6594c449d12c6f22b573ad835b5d66327aa9256"}
Oct 31 16:23:30.362603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.362500    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n" event={"ID":"ce328c5d-1aba-4298-99ef-8a06752141e6","Type":"ContainerStarted","Data":"9b7dea450f878c2ec201ad4aceefcc7cce89f41407c113be16222d6069feeff6"}
Oct 31 16:23:30.440698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.440664    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a5785d-612b-4b79-881d-077c9a1db489" path="/var/lib/kubelet/pods/54a5785d-612b-4b79-881d-077c9a1db489/volumes"
Oct 31 16:23:30.517947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:30.517898    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n" podStartSLOduration=2.5178835619999997 podStartE2EDuration="2.517883562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:30.516033179 +0000 UTC m=+1254.215962495" watchObservedRunningTime="2026-10-31 16:23:30.517883562 +0000 UTC m=+1254.217812925"
Oct 31 16:23:31.375377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:31.375336    4519 generic.go:356] "Generic (PLEG): container finished" podID="ce328c5d-1aba-4298-99ef-8a06752141e6" containerID="adc387ad69f85c4ce64edc08a6594c449d12c6f22b573ad835b5d66327aa9256" exitCode=0
Oct 31 16:23:31.375847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:31.375714    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n" event={"ID":"ce328c5d-1aba-4298-99ef-8a06752141e6","Type":"ContainerDied","Data":"adc387ad69f85c4ce64edc08a6594c449d12c6f22b573ad835b5d66327aa9256"}
Oct 31 16:23:32.556788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.556763    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:32.591377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.591350    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca\") pod \"ce328c5d-1aba-4298-99ef-8a06752141e6\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") "
Oct 31 16:23:32.591536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.591489    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca\") pod \"ce328c5d-1aba-4298-99ef-8a06752141e6\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") "
Oct 31 16:23:32.591536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.591524    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv\" (UniqueName: \"kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv\") pod \"ce328c5d-1aba-4298-99ef-8a06752141e6\" (UID: \"ce328c5d-1aba-4298-99ef-8a06752141e6\") "
Oct 31 16:23:32.591725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.591705    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca" pod "ce328c5d-1aba-4298-99ef-8a06752141e6" (UID: "ce328c5d-1aba-4298-99ef-8a06752141e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:32.592054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.592037    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:32.593543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.593515    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca" pod "ce328c5d-1aba-4298-99ef-8a06752141e6" (UID: "ce328c5d-1aba-4298-99ef-8a06752141e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:32.593638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.593597    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv" pod "ce328c5d-1aba-4298-99ef-8a06752141e6" (UID: "ce328c5d-1aba-4298-99ef-8a06752141e6"). InnerVolumeSpecName "kube-api-access-4wtjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:32.694056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.693971    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce328c5d-1aba-4298-99ef-8a06752141e6-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:32.694056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:32.694013    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wtjv\" (UniqueName: \"kubernetes.io/projected/ce328c5d-1aba-4298-99ef-8a06752141e6-kube-api-access-4wtjv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:33.396305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:33.396274    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"
Oct 31 16:23:33.396527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:33.396307    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n" event={"ID":"ce328c5d-1aba-4298-99ef-8a06752141e6","Type":"ContainerDied","Data":"9b7dea450f878c2ec201ad4aceefcc7cce89f41407c113be16222d6069feeff6"}
Oct 31 16:23:33.396527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:33.396378    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7dea450f878c2ec201ad4aceefcc7cce89f41407c113be16222d6069feeff6"
Oct 31 16:23:35.060049 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:35.060012    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice/crio-5f85850a05611b9ee1e435332a670a2dfee18aa455d68982ed68ca0b8656b85c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d0fa60_1708_423a_9e9c_818b8930977c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 16:23:35.169262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.169229    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"]
Oct 31 16:23:35.170021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.170001    4519 cpu_manager.go:414] "containerMap: removing container" podUID="25223c0f-3844-4822-b0cf-11ba4beff2cd" containerName="revoke-keys"
Oct 31 16:23:35.170021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.170021    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25223c0f-3844-4822-b0cf-11ba4beff2cd" containerName="revoke-keys"
Oct 31 16:23:35.170209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.170064    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ce328c5d-1aba-4298-99ef-8a06752141e6" containerName="revoke-keys"
Oct 31 16:23:35.170209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.170072    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce328c5d-1aba-4298-99ef-8a06752141e6" containerName="revoke-keys"
Oct 31 16:23:35.170330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.170315    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="25223c0f-3844-4822-b0cf-11ba4beff2cd" containerName="revoke-keys"
Oct 31 16:23:35.170388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.170338    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce328c5d-1aba-4298-99ef-8a06752141e6" containerName="revoke-keys"
Oct 31 16:23:35.174340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.174321    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.233632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.233563    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.233810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.233739    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59fq\" (UniqueName: \"kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.233810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.233795    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.238960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.238930    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"]
Oct 31 16:23:35.279509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.279474    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:23:35.279745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.279514    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:23:35.279745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.279605    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:23:35.279888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.279669    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:23:35.336044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.335966    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.336199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.336056    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59fq\" (UniqueName: \"kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.336199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.336082    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.336719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.336695    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.339280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.339252    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.389694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.389654    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59fq\" (UniqueName: \"kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq\") pod \"maas-api-revoke-sub-b7456051-hpcxd\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.490240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.490206    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:35.869522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:35.869080    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"]
Oct 31 16:23:35.872282 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:35.872255    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9e8cf8_5135_47b3_920c_1eca08513d13.slice/crio-1d3e41f182ee5c60156716a2d4444037ea3166df181e01cf415bc513e853c34d WatchSource:0}: Error finding container 1d3e41f182ee5c60156716a2d4444037ea3166df181e01cf415bc513e853c34d: Status 404 returned error can't find the container with id 1d3e41f182ee5c60156716a2d4444037ea3166df181e01cf415bc513e853c34d
Oct 31 16:23:36.399406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.399369    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"]
Oct 31 16:23:36.399803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.399696    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy" containerID="cri-o://cd464a24f1a0f9fa29904b5f5795150a00d990dbbe9b114c59324bced752dd23" gracePeriod=30
Oct 31 16:23:36.426985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.426944    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd" event={"ID":"ca9e8cf8-5135-47b3-920c-1eca08513d13","Type":"ContainerStarted","Data":"bfbc441758636bb2a15e64e728ece0a4807a6f23a964f5229bea6438c89692e9"}
Oct 31 16:23:36.427179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.427018    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd" event={"ID":"ca9e8cf8-5135-47b3-920c-1eca08513d13","Type":"ContainerStarted","Data":"1d3e41f182ee5c60156716a2d4444037ea3166df181e01cf415bc513e853c34d"}
Oct 31 16:23:36.446267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.446235    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"]
Oct 31 16:23:36.446487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.446448    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy" containerID="cri-o://bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc" gracePeriod=30
Oct 31 16:23:36.623317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.623261    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd" podStartSLOduration=1.623246959 podStartE2EDuration="1.623246959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:36.619061682 +0000 UTC m=+1260.318990995" watchObservedRunningTime="2026-10-31 16:23:36.623246959 +0000 UTC m=+1260.323176269"
Oct 31 16:23:36.728888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.728853    4519 scope.go:122] "RemoveContainer" containerID="8f32090c6cc0149db9c814603e19b5b2b0995412bb5a6dd3d80b467316fe6a87"
Oct 31 16:23:36.742519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.742498    4519 scope.go:122] "RemoveContainer" containerID="2a38a5e62185bd222af09dd37c4f8a6b5c7caf21726365f3d7b0214e30e798e6"
Oct 31 16:23:36.753090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.753067    4519 scope.go:122] "RemoveContainer" containerID="ebeb7673e80a37c0dded6bdee899dc1d0c7585935ed09431a49aefe4991c3ba6"
Oct 31 16:23:36.763600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.763578    4519 scope.go:122] "RemoveContainer" containerID="99b1a920b5adf22dbc3e2409829ee1fa43d5b29730bd3b34678211b5239ee39c"
Oct 31 16:23:36.773943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.773921    4519 scope.go:122] "RemoveContainer" containerID="df1bf362917cf228b6ab008a13022a2352f9c31d492cb025c4108fac2306806d"
Oct 31 16:23:36.857928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:36.857902    4519 scope.go:122] "RemoveContainer" containerID="d6611833883511d40baeb7daf22ba8b5a799cc40725c72dab7c5fb51e81c77fc"
Oct 31 16:23:37.030437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.030361    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"]
Oct 31 16:23:37.036419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.036393    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.062720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.062688    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2qq\" (UniqueName: \"kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.062894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.062817    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.063025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.063006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.087300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.087269    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"]
Oct 31 16:23:37.114078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.114047    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"]
Oct 31 16:23:37.118697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.118677    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.163997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.163961    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2qq\" (UniqueName: \"kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.164204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.164031    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.164204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.164109    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.164204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.164176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.164370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.164225    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfz94\" (UniqueName: \"kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.164483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.164439    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.165517 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.165487    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.167204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.167167    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.174433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.173594    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"]
Oct 31 16:23:37.218234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.218198    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2qq\" (UniqueName: \"kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq\") pod \"maas-api-revoke-sub-a13574d0-pgbqb\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.265917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.265883    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.266070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.265951    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.266070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.265995    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfz94\" (UniqueName: \"kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.266626 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.266605    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.268418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.268397    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.321600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.321522    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfz94\" (UniqueName: \"kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94\") pod \"maas-api-revoke-sub-268ebadf-gb2hb\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.351376 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.351348    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:37.437726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.437602    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:37.475213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.475029    4519 generic.go:356] "Generic (PLEG): container finished" podID="ca9e8cf8-5135-47b3-920c-1eca08513d13" containerID="bfbc441758636bb2a15e64e728ece0a4807a6f23a964f5229bea6438c89692e9" exitCode=0
Oct 31 16:23:37.475213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.475137    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd" event={"ID":"ca9e8cf8-5135-47b3-920c-1eca08513d13","Type":"ContainerDied","Data":"bfbc441758636bb2a15e64e728ece0a4807a6f23a964f5229bea6438c89692e9"}
Oct 31 16:23:37.812939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.812910    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"]
Oct 31 16:23:37.813788 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:37.813763    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c5339e_f5f9_443a_8c95_708d39f25b43.slice/crio-6c7df82a64b6408323d44abca407f86fb2431ec6d8d000654434992e74462304 WatchSource:0}: Error finding container 6c7df82a64b6408323d44abca407f86fb2431ec6d8d000654434992e74462304: Status 404 returned error can't find the container with id 6c7df82a64b6408323d44abca407f86fb2431ec6d8d000654434992e74462304
Oct 31 16:23:37.872921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:37.872553    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"]
Oct 31 16:23:37.875560 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:37.875526    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4343b8_1166_459c_b363_5d368bdfda3b.slice/crio-e5feb5c36740caf0063c3ff7d76cd9dd740914961048ee57969f3510e18797c2 WatchSource:0}: Error finding container e5feb5c36740caf0063c3ff7d76cd9dd740914961048ee57969f3510e18797c2: Status 404 returned error can't find the container with id e5feb5c36740caf0063c3ff7d76cd9dd740914961048ee57969f3510e18797c2
Oct 31 16:23:38.487275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.487245    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb" event={"ID":"6c4343b8-1166-459c-b363-5d368bdfda3b","Type":"ContainerStarted","Data":"acb667751afa3886f94bc0daf94b5c23e90b012d74219300234ffbaa514afb2a"}
Oct 31 16:23:38.487890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.487859    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb" event={"ID":"6c4343b8-1166-459c-b363-5d368bdfda3b","Type":"ContainerStarted","Data":"e5feb5c36740caf0063c3ff7d76cd9dd740914961048ee57969f3510e18797c2"}
Oct 31 16:23:38.489800 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.489774    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb" event={"ID":"55c5339e-f5f9-443a-8c95-708d39f25b43","Type":"ContainerStarted","Data":"d9c8bd51b3120a09e12752c6629046ba1cc50aa2552f70726e96e0456ee493d4"}
Oct 31 16:23:38.489923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.489821    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb" event={"ID":"55c5339e-f5f9-443a-8c95-708d39f25b43","Type":"ContainerStarted","Data":"6c7df82a64b6408323d44abca407f86fb2431ec6d8d000654434992e74462304"}
Oct 31 16:23:38.598215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.597559    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb" podStartSLOduration=1.597539641 podStartE2EDuration="1.597539641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:38.59558523 +0000 UTC m=+1262.295514542" watchObservedRunningTime="2026-10-31 16:23:38.597539641 +0000 UTC m=+1262.297468953"
Oct 31 16:23:38.706067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.706012    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb" podStartSLOduration=2.705996661 podStartE2EDuration="2.705996661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:38.704196931 +0000 UTC m=+1262.404126234" watchObservedRunningTime="2026-10-31 16:23:38.705996661 +0000 UTC m=+1262.405925972"
Oct 31 16:23:38.742067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.741450    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:38.815822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.815788    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq\" (UniqueName: \"kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq\") pod \"ca9e8cf8-5135-47b3-920c-1eca08513d13\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") "
Oct 31 16:23:38.815977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.815953    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca\") pod \"ca9e8cf8-5135-47b3-920c-1eca08513d13\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") "
Oct 31 16:23:38.816168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.816154    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca\") pod \"ca9e8cf8-5135-47b3-920c-1eca08513d13\" (UID: \"ca9e8cf8-5135-47b3-920c-1eca08513d13\") "
Oct 31 16:23:38.816518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.816442    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca" pod "ca9e8cf8-5135-47b3-920c-1eca08513d13" (UID: "ca9e8cf8-5135-47b3-920c-1eca08513d13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:38.816789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.816771    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:38.818066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.818039    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca" pod "ca9e8cf8-5135-47b3-920c-1eca08513d13" (UID: "ca9e8cf8-5135-47b3-920c-1eca08513d13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:38.818162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.818130    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq" pod "ca9e8cf8-5135-47b3-920c-1eca08513d13" (UID: "ca9e8cf8-5135-47b3-920c-1eca08513d13"). InnerVolumeSpecName "kube-api-access-v59fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:38.931970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.926813    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59fq\" (UniqueName: \"kubernetes.io/projected/ca9e8cf8-5135-47b3-920c-1eca08513d13-kube-api-access-v59fq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:38.931970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:38.926850    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca9e8cf8-5135-47b3-920c-1eca08513d13-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:39.334236 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.334151    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"]
Oct 31 16:23:39.334941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.334917    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ca9e8cf8-5135-47b3-920c-1eca08513d13" containerName="revoke-keys"
Oct 31 16:23:39.334941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.334935    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9e8cf8-5135-47b3-920c-1eca08513d13" containerName="revoke-keys"
Oct 31 16:23:39.335156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.335056    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9e8cf8-5135-47b3-920c-1eca08513d13" containerName="revoke-keys"
Oct 31 16:23:39.343538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.343507    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.390634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.390598    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"]
Oct 31 16:23:39.445552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.433862    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.445552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.433966    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.445552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.434130    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qlz\" (UniqueName: \"kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.499556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.499516    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"
Oct 31 16:23:39.499973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.499523    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd" event={"ID":"ca9e8cf8-5135-47b3-920c-1eca08513d13","Type":"ContainerDied","Data":"1d3e41f182ee5c60156716a2d4444037ea3166df181e01cf415bc513e853c34d"}
Oct 31 16:23:39.499973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.499693    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d3e41f182ee5c60156716a2d4444037ea3166df181e01cf415bc513e853c34d"
Oct 31 16:23:39.501432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.501405    4519 generic.go:356] "Generic (PLEG): container finished" podID="6c4343b8-1166-459c-b363-5d368bdfda3b" containerID="acb667751afa3886f94bc0daf94b5c23e90b012d74219300234ffbaa514afb2a" exitCode=0
Oct 31 16:23:39.501573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.501524    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb" event={"ID":"6c4343b8-1166-459c-b363-5d368bdfda3b","Type":"ContainerDied","Data":"acb667751afa3886f94bc0daf94b5c23e90b012d74219300234ffbaa514afb2a"}
Oct 31 16:23:39.506213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.506186    4519 generic.go:356] "Generic (PLEG): container finished" podID="55c5339e-f5f9-443a-8c95-708d39f25b43" containerID="d9c8bd51b3120a09e12752c6629046ba1cc50aa2552f70726e96e0456ee493d4" exitCode=0
Oct 31 16:23:39.506316 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.506276    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb" event={"ID":"55c5339e-f5f9-443a-8c95-708d39f25b43","Type":"ContainerDied","Data":"d9c8bd51b3120a09e12752c6629046ba1cc50aa2552f70726e96e0456ee493d4"}
Oct 31 16:23:39.563520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.558019    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qlz\" (UniqueName: \"kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.563520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.559406    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.563520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.559657    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.569673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.567324    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.573475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.569837    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.612394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.612317    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qlz\" (UniqueName: \"kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz\") pod \"maas-api-revoke-sub-7c0c9ac9-ldfnd\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:39.672533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:39.672497    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:40.116579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.116546    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"]
Oct 31 16:23:40.120023 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:23:40.119996    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5559663_048a_4e9b_9650_77def958c683.slice/crio-6db03f42e9072c258a2257f2bb94355509edc07cf9491b57f0c540a3d0740408 WatchSource:0}: Error finding container 6db03f42e9072c258a2257f2bb94355509edc07cf9491b57f0c540a3d0740408: Status 404 returned error can't find the container with id 6db03f42e9072c258a2257f2bb94355509edc07cf9491b57f0c540a3d0740408
Oct 31 16:23:40.518913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.518872    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd" event={"ID":"d5559663-048a-4e9b-9650-77def958c683","Type":"ContainerStarted","Data":"6db03f42e9072c258a2257f2bb94355509edc07cf9491b57f0c540a3d0740408"}
Oct 31 16:23:40.781537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.781507    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:40.785308 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.785281    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:40.894388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.894352    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca\") pod \"55c5339e-f5f9-443a-8c95-708d39f25b43\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") "
Oct 31 16:23:40.894589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.894415    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca\") pod \"55c5339e-f5f9-443a-8c95-708d39f25b43\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") "
Oct 31 16:23:40.894715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.894698    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq\" (UniqueName: \"kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq\") pod \"55c5339e-f5f9-443a-8c95-708d39f25b43\" (UID: \"55c5339e-f5f9-443a-8c95-708d39f25b43\") "
Oct 31 16:23:40.894780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.894762    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca\") pod \"6c4343b8-1166-459c-b363-5d368bdfda3b\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") "
Oct 31 16:23:40.894832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.894806    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca\") pod \"6c4343b8-1166-459c-b363-5d368bdfda3b\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") "
Oct 31 16:23:40.894926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.894913    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94\" (UniqueName: \"kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94\") pod \"6c4343b8-1166-459c-b363-5d368bdfda3b\" (UID: \"6c4343b8-1166-459c-b363-5d368bdfda3b\") "
Oct 31 16:23:40.897441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.897410    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca" pod "55c5339e-f5f9-443a-8c95-708d39f25b43" (UID: "55c5339e-f5f9-443a-8c95-708d39f25b43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:40.897623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.897530    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca" pod "6c4343b8-1166-459c-b363-5d368bdfda3b" (UID: "6c4343b8-1166-459c-b363-5d368bdfda3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:40.900005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.899977    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca" pod "55c5339e-f5f9-443a-8c95-708d39f25b43" (UID: "55c5339e-f5f9-443a-8c95-708d39f25b43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:40.900136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.900118    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq" pod "55c5339e-f5f9-443a-8c95-708d39f25b43" (UID: "55c5339e-f5f9-443a-8c95-708d39f25b43"). InnerVolumeSpecName "kube-api-access-7z2qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:40.900952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.900924    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca" pod "6c4343b8-1166-459c-b363-5d368bdfda3b" (UID: "6c4343b8-1166-459c-b363-5d368bdfda3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:40.901477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.901437    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94" pod "6c4343b8-1166-459c-b363-5d368bdfda3b" (UID: "6c4343b8-1166-459c-b363-5d368bdfda3b"). InnerVolumeSpecName "kube-api-access-tfz94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:40.997525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.997401    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:40.997525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.997432    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c5339e-f5f9-443a-8c95-708d39f25b43-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:40.997525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.997443    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z2qq\" (UniqueName: \"kubernetes.io/projected/55c5339e-f5f9-443a-8c95-708d39f25b43-kube-api-access-7z2qq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:40.997525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.997467    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:40.997525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.997499    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c4343b8-1166-459c-b363-5d368bdfda3b-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:40.997525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:40.997509    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfz94\" (UniqueName: \"kubernetes.io/projected/6c4343b8-1166-459c-b363-5d368bdfda3b-kube-api-access-tfz94\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.529592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.529565    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"
Oct 31 16:23:41.529957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.529594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb" event={"ID":"55c5339e-f5f9-443a-8c95-708d39f25b43","Type":"ContainerDied","Data":"6c7df82a64b6408323d44abca407f86fb2431ec6d8d000654434992e74462304"}
Oct 31 16:23:41.529957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.529654    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7df82a64b6408323d44abca407f86fb2431ec6d8d000654434992e74462304"
Oct 31 16:23:41.531417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.531395    4519 generic.go:356] "Generic (PLEG): container finished" podID="88e9db4a-ba99-4082-bc38-037581fe4449" containerID="cd464a24f1a0f9fa29904b5f5795150a00d990dbbe9b114c59324bced752dd23" exitCode=0
Oct 31 16:23:41.531564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.531503    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" event={"ID":"88e9db4a-ba99-4082-bc38-037581fe4449","Type":"ContainerDied","Data":"cd464a24f1a0f9fa29904b5f5795150a00d990dbbe9b114c59324bced752dd23"}
Oct 31 16:23:41.533310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.533288    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"
Oct 31 16:23:41.533491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.533438    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb" event={"ID":"6c4343b8-1166-459c-b363-5d368bdfda3b","Type":"ContainerDied","Data":"e5feb5c36740caf0063c3ff7d76cd9dd740914961048ee57969f3510e18797c2"}
Oct 31 16:23:41.533616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.533595    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5feb5c36740caf0063c3ff7d76cd9dd740914961048ee57969f3510e18797c2"
Oct 31 16:23:41.535477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.535427    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5559663-048a-4e9b-9650-77def958c683" containerID="ba15e4146ae344793688cfb99d589edf83dc1d4c0a2510419e7afda14987c50b" exitCode=0
Oct 31 16:23:41.535574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.535482    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd" event={"ID":"d5559663-048a-4e9b-9650-77def958c683","Type":"ContainerDied","Data":"ba15e4146ae344793688cfb99d589edf83dc1d4c0a2510419e7afda14987c50b"}
Oct 31 16:23:41.762904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.762876    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:23:41.793006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.792982    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:23:41.913559 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913423    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.913731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913554    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.913731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.913731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913650    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.913731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913701    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.913951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913732    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.913951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913763    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.913951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913850    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:41.913951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913921    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.914177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.913977    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.914177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914137    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.914285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914183    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.914285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.914285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914263    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.914440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914302    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.914440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914338    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.914440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914364    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.914440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914426    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket\") pod \"0e200262-0853-4f53-b7af-561ba9cfab51\" (UID: \"0e200262-0853-4f53-b7af-561ba9cfab51\") "
Oct 31 16:23:41.914665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914505    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy\") pod \"88e9db4a-ba99-4082-bc38-037581fe4449\" (UID: \"88e9db4a-ba99-4082-bc38-037581fe4449\") "
Oct 31 16:23:41.914783 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.914762    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915017    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915117    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915238    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915338    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915575    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915772    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.915910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.915840    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:41.916405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916380    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e200262-0853-4f53-b7af-561ba9cfab51-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916411    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e9db4a-ba99-4082-bc38-037581fe4449-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916425    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916437    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916469    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916483    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916495    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916508    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916522    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.916854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.916537    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:41.918204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.918141    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "kube-api-access-jqggg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:41.918204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.918190    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:23:41.918359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.918218    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:41.918619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.918593    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token" pod "0e200262-0853-4f53-b7af-561ba9cfab51" (UID: "0e200262-0853-4f53-b7af-561ba9cfab51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:41.918935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.918912    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:23:41.919409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.919382    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:41.919543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.919527    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "kube-api-access-nnrnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:41.919747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:41.919723    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy" pod "88e9db4a-ba99-4082-bc38-037581fe4449" (UID: "88e9db4a-ba99-4082-bc38-037581fe4449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:42.018212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018174    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e200262-0853-4f53-b7af-561ba9cfab51-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018214    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnrnk\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-kube-api-access-nnrnk\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018231    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e9db4a-ba99-4082-bc38-037581fe4449-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018246    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018260    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqggg\" (UniqueName: \"kubernetes.io/projected/0e200262-0853-4f53-b7af-561ba9cfab51-kube-api-access-jqggg\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018274    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e200262-0853-4f53-b7af-561ba9cfab51-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018287    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e9db4a-ba99-4082-bc38-037581fe4449-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.018410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.018301    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e9db4a-ba99-4082-bc38-037581fe4449-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.546163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.546128    4519 generic.go:356] "Generic (PLEG): container finished" podID="0e200262-0853-4f53-b7af-561ba9cfab51" containerID="bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc" exitCode=0
Oct 31 16:23:42.546612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.546194    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"
Oct 31 16:23:42.546612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.546215    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" event={"ID":"0e200262-0853-4f53-b7af-561ba9cfab51","Type":"ContainerDied","Data":"bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc"}
Oct 31 16:23:42.546612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.546280    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj" event={"ID":"0e200262-0853-4f53-b7af-561ba9cfab51","Type":"ContainerDied","Data":"99fa5299a0c9587641d816c914e975c9f3771e16ccd969a7d7e2e0e323a46c83"}
Oct 31 16:23:42.546612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.546306    4519 scope.go:122] "RemoveContainer" containerID="bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc"
Oct 31 16:23:42.548206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.548183    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"
Oct 31 16:23:42.548320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.548226    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n" event={"ID":"88e9db4a-ba99-4082-bc38-037581fe4449","Type":"ContainerDied","Data":"68684bca0755a4fb224f8776465fd6d1cd98575aff0925955d28c6afe402873e"}
Oct 31 16:23:42.562328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.562304    4519 scope.go:122] "RemoveContainer" containerID="bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc"
Oct 31 16:23:42.563018 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:42.562994    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc\": container with ID starting with bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc not found: ID does not exist" containerID="bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc"
Oct 31 16:23:42.563112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.563025    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc"} err="failed to get container status \"bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc\": rpc error: code = NotFound desc = could not find container \"bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc\": container with ID starting with bebdda8701b4843db56c7623f02c0e05b92fa3412b3bfd6d9038ae1be60774fc not found: ID does not exist"
Oct 31 16:23:42.563112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.563049    4519 scope.go:122] "RemoveContainer" containerID="cd464a24f1a0f9fa29904b5f5795150a00d990dbbe9b114c59324bced752dd23"
Oct 31 16:23:42.728716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.728678    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:42.780005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.779969    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"]
Oct 31 16:23:42.824873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.824839    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dgfgj"]
Oct 31 16:23:42.829948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.829909    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca\") pod \"d5559663-048a-4e9b-9650-77def958c683\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") "
Oct 31 16:23:42.830130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.830114    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz\" (UniqueName: \"kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz\") pod \"d5559663-048a-4e9b-9650-77def958c683\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") "
Oct 31 16:23:42.830252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.830231    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca" pod "d5559663-048a-4e9b-9650-77def958c683" (UID: "d5559663-048a-4e9b-9650-77def958c683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:42.830340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.830232    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca\") pod \"d5559663-048a-4e9b-9650-77def958c683\" (UID: \"d5559663-048a-4e9b-9650-77def958c683\") "
Oct 31 16:23:42.830862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.830840    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5559663-048a-4e9b-9650-77def958c683-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.832254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.832228    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca" pod "d5559663-048a-4e9b-9650-77def958c683" (UID: "d5559663-048a-4e9b-9650-77def958c683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:23:42.832345 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.832297    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz" pod "d5559663-048a-4e9b-9650-77def958c683" (UID: "d5559663-048a-4e9b-9650-77def958c683"). InnerVolumeSpecName "kube-api-access-b2qlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:42.886725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.886653    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"]
Oct 31 16:23:42.891867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.891841    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fqh2n"]
Oct 31 16:23:42.932111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.932070    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qlz\" (UniqueName: \"kubernetes.io/projected/d5559663-048a-4e9b-9650-77def958c683-kube-api-access-b2qlz\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:42.932111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:42.932099    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5559663-048a-4e9b-9650-77def958c683-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:43.619596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:43.605594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd" event={"ID":"d5559663-048a-4e9b-9650-77def958c683","Type":"ContainerDied","Data":"6db03f42e9072c258a2257f2bb94355509edc07cf9491b57f0c540a3d0740408"}
Oct 31 16:23:43.619596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:43.605669    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db03f42e9072c258a2257f2bb94355509edc07cf9491b57f0c540a3d0740408"
Oct 31 16:23:43.619596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:43.605754    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"
Oct 31 16:23:44.442244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:44.442210    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" path="/var/lib/kubelet/pods/0e200262-0853-4f53-b7af-561ba9cfab51/volumes"
Oct 31 16:23:44.442884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:44.442865    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" path="/var/lib/kubelet/pods/88e9db4a-ba99-4082-bc38-037581fe4449/volumes"
Oct 31 16:23:54.005734 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:23:54.005611    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:23:54.807266 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:54.807234    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:23:55.379799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.379758    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"]
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382606    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6c4343b8-1166-459c-b363-5d368bdfda3b" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382636    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4343b8-1166-459c-b363-5d368bdfda3b" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382652    4519 cpu_manager.go:414] "containerMap: removing container" podUID="55c5339e-f5f9-443a-8c95-708d39f25b43" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382659    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c5339e-f5f9-443a-8c95-708d39f25b43" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382702    4519 cpu_manager.go:414] "containerMap: removing container" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382710    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382742    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382750    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382766    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5559663-048a-4e9b-9650-77def958c683" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382775    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5559663-048a-4e9b-9650-77def958c683" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382946    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4343b8-1166-459c-b363-5d368bdfda3b" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382962    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c5339e-f5f9-443a-8c95-708d39f25b43" containerName="revoke-keys"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382978    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e9db4a-ba99-4082-bc38-037581fe4449" containerName="istio-proxy"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.382993    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e200262-0853-4f53-b7af-561ba9cfab51" containerName="istio-proxy"
Oct 31 16:23:55.384190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.383003    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5559663-048a-4e9b-9650-77def958c683" containerName="revoke-keys"
Oct 31 16:23:55.396566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.390871    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.480341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.480303    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"]
Oct 31 16:23:55.485644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.484288    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:23:55.485644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.484390    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:23:55.485644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.484553    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:23:55.485644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.484627    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:23:55.519622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.519586    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.520159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.520131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.520264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.520190    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgcn\" (UniqueName: \"kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.622846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.622806    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.623068 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.622873    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgcn\" (UniqueName: \"kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.623068 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.623042    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.624083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.624049    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.628146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.628112    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.683521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.682531    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgcn\" (UniqueName: \"kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn\") pod \"maas-api-revoke-sub-6569ee9e-c6kll\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:55.763422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:55.763116    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:23:57.554648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.551839    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.554648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.552140    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:23:57.554648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.552761    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.554648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.552958    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.607484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.605894    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.635482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.611891    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"]
Oct 31 16:23:57.680575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.667620    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:23:57.714601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.702674    4519 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:57.714601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.702758    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.764490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.744509    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.919726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.919617    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.949981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:57.949930    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.001952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.001908    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.035546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.002223    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.096970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.096924    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.332485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.331803    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll" event={"ID":"1c819957-a750-4960-a8bb-fcb5f0da745f","Type":"ContainerStarted","Data":"75bb4535d648b30260ff653fb56d084f476ee77fe5df3be9671d86dc68d9991c"}
Oct 31 16:23:58.439095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.415595    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.552736    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.576716    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.582200    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.582514    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.609813    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.613552    4519 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.613612    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.721605    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:58.784554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.769440    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.858234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:58.849860    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.035538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.034742    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.037149    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054564    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054658    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054715    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054768    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054823    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054882    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054934    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.054984    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:23:59.059479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.055035    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:23:59.111480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.101691    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:59.111480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.101760    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.230480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.199441    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.255480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.238612    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:23:59.335535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.330281    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:23:59.335535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.331379    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:23:59.399549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.384847    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:23:59.399549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.393966    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 31 16:23:59.399549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.394272    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:23:59.587544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.578805    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:23:59.587544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.579050    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:23:59.622513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.590682    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.622513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.590933    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:23:59.624051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.624011    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.847888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.847791    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-nxxwv" podUID="baf1235e-381e-406c-be7c-c19059168c8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.986480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:23:59.967916    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.096480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.052085    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.096480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.064624    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Oct 31 16:24:00.628026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.167338    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:24:00.708341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.175840    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.729063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.256522    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.730294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.333664    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.798508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.739592    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:24:00.863482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.740237    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:24:00.991491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.740288    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.740308    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.767211    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:24:00.841681    4519 logging.go:55] [core] [Channel #1793 SubChannel #1794]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: operation was canceled"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.895195    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2tnln" podUID="7f00984f-32ff-4b00-ae35-5372d3fd4ac3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.915063    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:00.915881    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.022497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.013766    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:24:01.063008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.062961    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:24:01.178481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.118021    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll" event={"ID":"1c819957-a750-4960-a8bb-fcb5f0da745f","Type":"ContainerStarted","Data":"cab09241317d9f435d4d9668dcdfb43b84a5eb6a7a711874aa7471307b8ae744"}
Oct 31 16:24:01.724488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.708921    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.738122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.736573    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.790718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.741754    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.865483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:01.842617    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.077771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.062849    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.077771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.063400    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.077771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.066122    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.084501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.082575    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.084501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.082961    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.084501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.083234    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.084501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.083501    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.128706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.128660    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 16:24:02.130429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.130383    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.214289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.214252    4519 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:02.214610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.214575    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.265940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.265903    4519 generic.go:356] "Generic (PLEG): container finished" podID="1c819957-a750-4960-a8bb-fcb5f0da745f" containerID="cab09241317d9f435d4d9668dcdfb43b84a5eb6a7a711874aa7471307b8ae744" exitCode=0
Oct 31 16:24:02.266319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.266293    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll" event={"ID":"1c819957-a750-4960-a8bb-fcb5f0da745f","Type":"ContainerDied","Data":"cab09241317d9f435d4d9668dcdfb43b84a5eb6a7a711874aa7471307b8ae744"}
Oct 31 16:24:02.730491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.730430    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.750516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.743762    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.763491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.743801    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.855412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:02.853168    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.134535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.107690    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.243480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.142357    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.276480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.142395    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.155642    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.155696    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.155720    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.155744    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.157285    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.174096    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.313767    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:03.514115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.484440    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.668555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.628667    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.766484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.756576    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.766484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.756996    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.860796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:03.859629    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:24:04.037792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.037762    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:24:04.043525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.043479    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:04.063511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.063484    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:24:04.063771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.063757    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:24:04.084412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.084377    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:24:04.084731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.084717    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:24:04.084951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.084939    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:24:04.085143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.085132    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:24:04.104919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.100664    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:24:04.104919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.100739    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:24:04.104919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.100792    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:24:04.108146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.106120    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:24:04.124605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.124577    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:24:04.125480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.125337    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:24:04.251886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.251855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca\") pod \"1c819957-a750-4960-a8bb-fcb5f0da745f\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") "
Oct 31 16:24:04.252255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.251903    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca\") pod \"1c819957-a750-4960-a8bb-fcb5f0da745f\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") "
Oct 31 16:24:04.252255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.251988    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn\" (UniqueName: \"kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn\") pod \"1c819957-a750-4960-a8bb-fcb5f0da745f\" (UID: \"1c819957-a750-4960-a8bb-fcb5f0da745f\") "
Oct 31 16:24:04.252335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.252262    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca" pod "1c819957-a750-4960-a8bb-fcb5f0da745f" (UID: "1c819957-a750-4960-a8bb-fcb5f0da745f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:24:04.253618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.253591    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:24:04.254314 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.254289    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca" pod "1c819957-a750-4960-a8bb-fcb5f0da745f" (UID: "1c819957-a750-4960-a8bb-fcb5f0da745f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:24:04.254431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.254297    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn" pod "1c819957-a750-4960-a8bb-fcb5f0da745f" (UID: "1c819957-a750-4960-a8bb-fcb5f0da745f"). InnerVolumeSpecName "kube-api-access-pxgcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:24:04.355584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.355504    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c819957-a750-4960-a8bb-fcb5f0da745f-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:24:04.355584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.355529    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgcn\" (UniqueName: \"kubernetes.io/projected/1c819957-a750-4960-a8bb-fcb5f0da745f-kube-api-access-pxgcn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:24:04.409566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.409535    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"
Oct 31 16:24:04.409755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.409594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll" event={"ID":"1c819957-a750-4960-a8bb-fcb5f0da745f","Type":"ContainerDied","Data":"75bb4535d648b30260ff653fb56d084f476ee77fe5df3be9671d86dc68d9991c"}
Oct 31 16:24:04.409755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:04.409653    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75bb4535d648b30260ff653fb56d084f476ee77fe5df3be9671d86dc68d9991c"
Oct 31 16:24:24.009093 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:24:24.009011    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:24:36.925884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.925844    4519 scope.go:122] "RemoveContainer" containerID="c62ca0eb34d3e58085cff86a971a35f9308c3f2a5ccd126c57ba2e934fb7e63c"
Oct 31 16:24:36.939068 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.939043    4519 scope.go:122] "RemoveContainer" containerID="22632dbdec7783f9c32a71a0062b080b0b8012295a7bb68c68eb5dd143c8be13"
Oct 31 16:24:36.949936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.949910    4519 scope.go:122] "RemoveContainer" containerID="b1836f7318077312072f24c4a7222c7bdc71d05c09ec59d8fc1842cc1a72e263"
Oct 31 16:24:36.965831 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.965809    4519 scope.go:122] "RemoveContainer" containerID="4c8beeaad1f43064d9a59a62072726bff326f793c8f77a100ea84b89d1555cb1"
Oct 31 16:24:36.977740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.977717    4519 scope.go:122] "RemoveContainer" containerID="66cb48e4a08189c967873b14346188b03038dc8e03e3b155b5d0c052eef00d66"
Oct 31 16:24:36.987888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.987867    4519 scope.go:122] "RemoveContainer" containerID="8b418c560fc4c3c261b1f944c4286b375d6b9ae1735ca0a30143bab7a55a0ceb"
Oct 31 16:24:36.997745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:36.997683    4519 scope.go:122] "RemoveContainer" containerID="30d6b371da8eb61a1036c703bcc9ff577fa3c1ca2d7effe01edd18f3a42ae86d"
Oct 31 16:24:37.010224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:37.010202    4519 scope.go:122] "RemoveContainer" containerID="d384b5c902b2ba9ef0fad66910287ce6ae7f0bb7214f79d9e15eef2a03920f26"
Oct 31 16:24:37.028379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:37.028188    4519 scope.go:122] "RemoveContainer" containerID="9853e5f90c02447bcc8253de566199dffca46508b096186299b696553c24b969"
Oct 31 16:24:37.038814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:37.038786    4519 scope.go:122] "RemoveContainer" containerID="8a69d4a51a1f197dfc4e1eb81d7b249a382e348121af0f10259d0da93368c366"
Oct 31 16:24:37.051782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:37.051757    4519 scope.go:122] "RemoveContainer" containerID="cab01d6dce518aae6ddcbd7ccdafbc533c71b12308717c9115983ace2ba41583"
Oct 31 16:24:37.067843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:37.067819    4519 scope.go:122] "RemoveContainer" containerID="1d8397dbdb947479875f4c0b4db51d5d0fae1debf97b538dea9ebe300650eb82"
Oct 31 16:24:38.590475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:38.586554    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:38.600612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:38.600570    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:24:38.626567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:38.626521    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:38.833685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:38.833636    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:38.834403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:38.834369    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 16:24:38.894063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:38.886564    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.055501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.040666    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.055501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.040931    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:24:39.055501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.041396    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 31 16:24:39.071901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.071859    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded"
Oct 31 16:24:39.072535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.072503    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 16:24:39.146817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.146726    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.249580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.249533    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.646599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.646541    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 31 16:24:39.670491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.651281    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.670491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.651312    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Oct 31 16:24:39.670491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.652987    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:24:39.670491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.653050    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:24:39.719224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.719180    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.829690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.814545    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.835490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.831140    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:24:39.851323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.851275    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:24:39.870477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.859060    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:24:39.892070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.892028    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.892980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:39.892953    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:24:40.047479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.045655    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.052548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.047878    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:24:40.090479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.072829    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.090479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.074159    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.090479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.074599    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:24:40.090479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.074662    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:24:40.147975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.147920    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.168572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.154821    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:24:40.250086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.250041    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.253020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.252994    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:24:40.388691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.387064    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:24:40.404617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.404137    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:24:40.433555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.433525    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:24:40.689109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.689029    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:24:40.689729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.689699    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:24:40.689877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.689867    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:24:40.690003 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.689994    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:24:40.690128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.690118    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:24:40.690290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.690268    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn"
Oct 31 16:24:40.691212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:24:40.691188    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:24:54.005289 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:24:54.005248    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:25:07.182628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.182594    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"]
Oct 31 16:25:07.183282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.183267    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1c819957-a750-4960-a8bb-fcb5f0da745f" containerName="revoke-keys"
Oct 31 16:25:07.183282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.183282    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c819957-a750-4960-a8bb-fcb5f0da745f" containerName="revoke-keys"
Oct 31 16:25:07.183421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.183412    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c819957-a750-4960-a8bb-fcb5f0da745f" containerName="revoke-keys"
Oct 31 16:25:07.187297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.187279    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.289601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.289570    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"]
Oct 31 16:25:07.296850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.296821    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.296892    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.296995    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.297041    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.297276    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.297370    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.297435    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.297531    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rrh\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.297591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.297577    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.399687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.399651    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.399977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.399948    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400006    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400094    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rrh\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400136    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400195    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400264    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400305    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400333    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.400941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400915    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.401084 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.400972    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.401475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.401439    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.401606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.401548    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.401713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.401617    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.406243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.406219    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.406649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.406625    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.455233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.455161    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.455398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.455378    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rrh\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tc5qb\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.501444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.501413    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:07.935943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.935914    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"]
Oct 31 16:25:07.938980 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:07.938951    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90823085_6810_4c40_a994_1f59f903032f.slice/crio-9cf9002ad83fff7220c401a76c871feea2c6d1090bc3bc8ebc7c93e2b3800963 WatchSource:0}: Error finding container 9cf9002ad83fff7220c401a76c871feea2c6d1090bc3bc8ebc7c93e2b3800963: Status 404 returned error can't find the container with id 9cf9002ad83fff7220c401a76c871feea2c6d1090bc3bc8ebc7c93e2b3800963
Oct 31 16:25:07.941520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.941485    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:07.941589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.941551    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:07.941645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:07.941615    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:08.484825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:08.484192    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" event={"ID":"90823085-6810-4c40-a994-1f59f903032f","Type":"ContainerStarted","Data":"023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3"}
Oct 31 16:25:08.484825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:08.484266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" event={"ID":"90823085-6810-4c40-a994-1f59f903032f","Type":"ContainerStarted","Data":"9cf9002ad83fff7220c401a76c871feea2c6d1090bc3bc8ebc7c93e2b3800963"}
Oct 31 16:25:08.502553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:08.502521    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:08.600404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:08.600327    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podStartSLOduration=1.600308871 podStartE2EDuration="1.600308871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:08.598060973 +0000 UTC m=+1352.297990285" watchObservedRunningTime="2026-10-31 16:25:08.600308871 +0000 UTC m=+1352.300238187"
Oct 31 16:25:09.503949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:09.503915    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:09.504355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:09.503965    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:10.503301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:10.503265    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:10.503523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:10.503323    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:11.502366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:11.502330    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:11.502863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:11.502393    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:12.502696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:12.502659    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:12.503219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:12.502721    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:13.501919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:13.501885    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:13.502774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:13.501942    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:14.502430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:14.502393    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:14.502626 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:14.502467    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:15.501903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:15.501864    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 16:25:15.502460 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:15.502429    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 16:25:16.506204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:16.506177    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:16.506675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:16.506657    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:16.507287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:16.507270    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:25:21.305501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:21.305445    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:25:22.249183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.249141    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"]
Oct 31 16:25:22.255770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.255747    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.295877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.295847    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"]
Oct 31 16:25:22.300673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.300654    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349001    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"]
Oct 31 16:25:22.349435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349162    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349203    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349306    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.349435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349347    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.349435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349429    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349689    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcpm\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349750    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349777    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349842    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.349907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349863    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.349974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.349915    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.350072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350045    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.350161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350141    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.350273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350256    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.350413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.350492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350476    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdxk\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.350552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350528    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.350609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.350563    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.451997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.451959    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452025    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452065    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452380    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452427    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452497    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452521    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452554    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcpm\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452579    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452615    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452663    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452695    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.452728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452726    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"]
Oct 31 16:25:22.453305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452776    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.452823    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453115    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.453305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453215    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453316    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.453574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453374    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453407    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453466    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.453793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453581    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453655    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdxk\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453676    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.453793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.453736    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.454058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.454028    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.454410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.454385    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.454541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.454432    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.454898 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.454875    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.455757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.455732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.456621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.456593    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.457700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.457678    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.457826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.457719    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.506356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.506281    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.507938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.507906    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.510706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.510683    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcpm\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mb94r\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:22.542513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.541509    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdxk\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zgr9j\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.571587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.571551    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:22.615866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:22.615658    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:23.044268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.044186    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"]
Oct 31 16:25:23.082804 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:23.082767    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8019d988_49e9_448b_a12a_ad8b58514934.slice/crio-c09a0c4994e83c6ec393be560e1adf3e771100abab2b6eff51110a750e97db8e WatchSource:0}: Error finding container c09a0c4994e83c6ec393be560e1adf3e771100abab2b6eff51110a750e97db8e: Status 404 returned error can't find the container with id c09a0c4994e83c6ec393be560e1adf3e771100abab2b6eff51110a750e97db8e
Oct 31 16:25:23.085291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.085252    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:23.085380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.085345    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:23.085438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.085397    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:23.095033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.095007    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"]
Oct 31 16:25:23.100143 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:23.100114    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846e224_aeeb_4612_bb4b_e66f8366c1dc.slice/crio-2b9a15e942c4f25f8e6add1082e371fa473460d80b342ef38866639fd8f432da WatchSource:0}: Error finding container 2b9a15e942c4f25f8e6add1082e371fa473460d80b342ef38866639fd8f432da: Status 404 returned error can't find the container with id 2b9a15e942c4f25f8e6add1082e371fa473460d80b342ef38866639fd8f432da
Oct 31 16:25:23.102817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.102785    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:23.102894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.102849    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:23.102894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.102878    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:23.630410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.630369    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" event={"ID":"9846e224-aeeb-4612-bb4b-e66f8366c1dc","Type":"ContainerStarted","Data":"7587a643026f6ca5459d9efed901916005d9d41444cea0c583cd10212c8fb2c9"}
Oct 31 16:25:23.630958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.630488    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" event={"ID":"9846e224-aeeb-4612-bb4b-e66f8366c1dc","Type":"ContainerStarted","Data":"2b9a15e942c4f25f8e6add1082e371fa473460d80b342ef38866639fd8f432da"}
Oct 31 16:25:23.633186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.633154    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" event={"ID":"8019d988-49e9-448b-a12a-ad8b58514934","Type":"ContainerStarted","Data":"2eac6679292e49b8e3619d5d5a4513ac0bb3d037cf81a2597304f00f4abff0b8"}
Oct 31 16:25:23.633327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.633215    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" event={"ID":"8019d988-49e9-448b-a12a-ad8b58514934","Type":"ContainerStarted","Data":"c09a0c4994e83c6ec393be560e1adf3e771100abab2b6eff51110a750e97db8e"}
Oct 31 16:25:23.769474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.765869    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podStartSLOduration=1.765844563 podStartE2EDuration="1.765844563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:23.750103624 +0000 UTC m=+1367.450032948" watchObservedRunningTime="2026-10-31 16:25:23.765844563 +0000 UTC m=+1367.465773877"
Oct 31 16:25:23.916541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:23.902119    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podStartSLOduration=1.902092996 podStartE2EDuration="1.902092996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:23.863943052 +0000 UTC m=+1367.563872365" watchObservedRunningTime="2026-10-31 16:25:23.902092996 +0000 UTC m=+1367.602022308"
Oct 31 16:25:24.038749 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:25:24.038712    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:25:24.582480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:24.573804    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:24.608479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:24.597990    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:24.608479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:24.598059    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:24.622047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:24.619265    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:24.626418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:24.624576    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:24.626418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:24.624633    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:25.574890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:25.574853    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:25.579476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:25.575515    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:25.621496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:25.617358    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:25.621496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:25.617415    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:26.588938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:26.588899    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:26.589578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:26.589538    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:26.618116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:26.618078    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:26.618413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:26.618385    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:27.573955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:27.573913    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:27.574263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:27.574234    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:27.628475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:27.622665    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:27.628475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:27.622736    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:28.602500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:28.589882    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:28.602835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:28.602781    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:28.654648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:28.654609    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:28.655146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:28.654669    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:29.572181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:29.572138    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:29.572370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:29.572213    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:29.618106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:29.618055    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:29.618307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:29.618118    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:30.536075 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.536041    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"]
Oct 31 16:25:30.540928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.540897    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.572902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.572866    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zgr9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 31 16:25:30.573043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.572932    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 31 16:25:30.586306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586268    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.586486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586313    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.586575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586525    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.586575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586560    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.586730 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586703    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.586876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586857    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.586924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.586908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.587042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.587009    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.587176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.587088    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gw5\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.590652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.590620    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-caa664e1-openshift-default-dockercfg-cbxhc\""
Oct 31 16:25:30.616820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.616791    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:25:30.616991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.616842    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:25:30.650063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.650009    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"]
Oct 31 16:25:30.674474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.666281    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"]
Oct 31 16:25:30.674474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.672685    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.690001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.689965    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690022    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690138    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690171    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690252    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690333    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.690434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690418    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690480    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.690540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690519    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690584    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690630    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gw5\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.690681 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.690659    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29kfj\" (UniqueName: \"kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.691679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.691651    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.698520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.693437    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.698520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.693740    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.698520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.694916    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.698520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.695174    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.698520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.696258    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.698865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.698538    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.717514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.717448    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"]
Oct 31 16:25:30.724820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.724791    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:25:30.725275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.725252    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:25:30.758381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.758292    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.779275    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gw5\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.791178    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.791326    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.792323    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.792409    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.792495    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29kfj\" (UniqueName: \"kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.794586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.793892    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.798526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.796557    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.846367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.846323    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29kfj\" (UniqueName: \"kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj\") pod \"maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:30.859274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:30.859240    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:31.001037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.000996    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:31.311972 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:31.311938    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb282c03_372f_4370_a230_ddc2a1ac3b75.slice/crio-f837709536aa650ed263772fccd926ba43fe8fd323b476db681078e20b8816c8 WatchSource:0}: Error finding container f837709536aa650ed263772fccd926ba43fe8fd323b476db681078e20b8816c8: Status 404 returned error can't find the container with id f837709536aa650ed263772fccd926ba43fe8fd323b476db681078e20b8816c8
Oct 31 16:25:31.317625 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.317580    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:31.317888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.317859    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:31.318125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.318098    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:31.332787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.332755    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"]
Oct 31 16:25:31.406440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.406396    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"]
Oct 31 16:25:31.802485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.801275    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl" event={"ID":"13856c23-ff51-41a7-b150-229f4196297b","Type":"ContainerStarted","Data":"0410f66fa04521f2c6f7c8b83035315c5536d6ff36481e6658c3817ff029b005"}
Oct 31 16:25:31.805552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.803627    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" event={"ID":"db282c03-372f-4370-a230-ddc2a1ac3b75","Type":"ContainerStarted","Data":"f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566"}
Oct 31 16:25:31.805552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.803692    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" event={"ID":"db282c03-372f-4370-a230-ddc2a1ac3b75","Type":"ContainerStarted","Data":"f837709536aa650ed263772fccd926ba43fe8fd323b476db681078e20b8816c8"}
Oct 31 16:25:31.859810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.859737    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:31.924486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:31.924261    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podStartSLOduration=1.924242334 podStartE2EDuration="1.924242334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:31.916798512 +0000 UTC m=+1375.616727825" watchObservedRunningTime="2026-10-31 16:25:31.924242334 +0000 UTC m=+1375.624171648"
Oct 31 16:25:32.259592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.257599    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:32.259592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.258728    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:32.259832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.259734    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:25:32.294845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.294813    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:32.296633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.296611    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:32.301064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.301042    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:25:32.817083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.817045    4519 generic.go:356] "Generic (PLEG): container finished" podID="13856c23-ff51-41a7-b150-229f4196297b" containerID="3b24dfd2e7e78994f08431887782c27874e265fbebaeb9a07fe924afa6cadfc8" exitCode=0
Oct 31 16:25:32.817772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.817122    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl" event={"ID":"13856c23-ff51-41a7-b150-229f4196297b","Type":"ContainerDied","Data":"3b24dfd2e7e78994f08431887782c27874e265fbebaeb9a07fe924afa6cadfc8"}
Oct 31 16:25:32.862040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.862002    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 16:25:32.862253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:32.862062    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 16:25:33.860418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:33.860385    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 16:25:33.860814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:33.860439    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 16:25:34.027577 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.027552    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:34.100406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.100280    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca\") pod \"13856c23-ff51-41a7-b150-229f4196297b\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") "
Oct 31 16:25:34.100406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.100380    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca\") pod \"13856c23-ff51-41a7-b150-229f4196297b\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") "
Oct 31 16:25:34.100745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.100560    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj\" (UniqueName: \"kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj\") pod \"13856c23-ff51-41a7-b150-229f4196297b\" (UID: \"13856c23-ff51-41a7-b150-229f4196297b\") "
Oct 31 16:25:34.101037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.100986    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca" pod "13856c23-ff51-41a7-b150-229f4196297b" (UID: "13856c23-ff51-41a7-b150-229f4196297b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:34.101813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.101784    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13856c23-ff51-41a7-b150-229f4196297b-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:34.105637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.104678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca" pod "13856c23-ff51-41a7-b150-229f4196297b" (UID: "13856c23-ff51-41a7-b150-229f4196297b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:25:34.107670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.107643    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj" pod "13856c23-ff51-41a7-b150-229f4196297b" (UID: "13856c23-ff51-41a7-b150-229f4196297b"). InnerVolumeSpecName "kube-api-access-29kfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:34.206507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.203567    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13856c23-ff51-41a7-b150-229f4196297b-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:34.206507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.203606    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29kfj\" (UniqueName: \"kubernetes.io/projected/13856c23-ff51-41a7-b150-229f4196297b-kube-api-access-29kfj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:34.834802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.834766    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl" event={"ID":"13856c23-ff51-41a7-b150-229f4196297b","Type":"ContainerDied","Data":"0410f66fa04521f2c6f7c8b83035315c5536d6ff36481e6658c3817ff029b005"}
Oct 31 16:25:34.834802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.834804    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"
Oct 31 16:25:34.835019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.834816    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0410f66fa04521f2c6f7c8b83035315c5536d6ff36481e6658c3817ff029b005"
Oct 31 16:25:34.859871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.859841    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 16:25:34.860002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:34.859891    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 16:25:35.864518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:35.864476    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 16:25:35.865050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:35.864562    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 16:25:36.859805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.859766    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 16:25:36.860026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.859831    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 16:25:36.898808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.898770    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"]
Oct 31 16:25:36.899905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.899880    4519 cpu_manager.go:414] "containerMap: removing container" podUID="13856c23-ff51-41a7-b150-229f4196297b" containerName="revoke-keys"
Oct 31 16:25:36.899905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.899903    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13856c23-ff51-41a7-b150-229f4196297b" containerName="revoke-keys"
Oct 31 16:25:36.900149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.900135    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="13856c23-ff51-41a7-b150-229f4196297b" containerName="revoke-keys"
Oct 31 16:25:36.904982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.904961    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:36.956076 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.956044    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"]
Oct 31 16:25:36.982481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.961411    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:25:36.995479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.994982    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:25:36.995479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.995110    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:25:36.995479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:36.995167    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:25:37.056726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.056694    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.056917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.056750    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.056964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.056919    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksv5p\" (UniqueName: \"kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.158697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.158614    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.158697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.158688    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.158879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.158795    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksv5p\" (UniqueName: \"kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.159351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.159329    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.161008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.160985    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.212946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.212919    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksv5p\" (UniqueName: \"kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p\") pod \"maas-api-revoke-sub-20212d6c-zmmmh\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.219934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.219907    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:37.441914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.440990    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"]
Oct 31 16:25:37.447153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.447131    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.499044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.499010    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"]
Oct 31 16:25:37.565126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.565089    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s946s\" (UniqueName: \"kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.565291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.565198    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.565335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.565312    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.670851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.670819    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s946s\" (UniqueName: \"kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.671026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.670901    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.671026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.670963    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.671764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.671746    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.674343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.674259    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.681901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.681876    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"]
Oct 31 16:25:37.682311 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:37.682288    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0920931b_99c7_4ae9_9a9c_c3cef0dc8f32.slice/crio-4effe0c50b00844a43fc8332d34f2645cf92b9425ec7bd8d44abc5c21b46007f WatchSource:0}: Error finding container 4effe0c50b00844a43fc8332d34f2645cf92b9425ec7bd8d44abc5c21b46007f: Status 404 returned error can't find the container with id 4effe0c50b00844a43fc8332d34f2645cf92b9425ec7bd8d44abc5c21b46007f
Oct 31 16:25:37.725380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.725352    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s946s\" (UniqueName: \"kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s\") pod \"maas-api-revoke-keys-e2e-ait-a08562e5-ch6th\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.762194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.762158    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:37.879823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.879790    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:37.882538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.882518    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:37.884537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.884402    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:25:37.887378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:37.887353    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh" event={"ID":"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32","Type":"ContainerStarted","Data":"4effe0c50b00844a43fc8332d34f2645cf92b9425ec7bd8d44abc5c21b46007f"}
Oct 31 16:25:38.169385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:38.169310    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"]
Oct 31 16:25:38.172892 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:38.172845    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573809fb_7fa4_459e_8457_127ec7fecd38.slice/crio-28b6148886dadea1194b9c33f91466ee4a09b2a3dd545e4ffe12920bc33b4acc WatchSource:0}: Error finding container 28b6148886dadea1194b9c33f91466ee4a09b2a3dd545e4ffe12920bc33b4acc: Status 404 returned error can't find the container with id 28b6148886dadea1194b9c33f91466ee4a09b2a3dd545e4ffe12920bc33b4acc
Oct 31 16:25:38.900538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:38.899524    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th" event={"ID":"573809fb-7fa4-459e-8457-127ec7fecd38","Type":"ContainerStarted","Data":"b449aacd8e7c773d1cef8aed072b6078c177915c7984f172bf3e54b1f38cc959"}
Oct 31 16:25:38.900538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:38.899621    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th" event={"ID":"573809fb-7fa4-459e-8457-127ec7fecd38","Type":"ContainerStarted","Data":"28b6148886dadea1194b9c33f91466ee4a09b2a3dd545e4ffe12920bc33b4acc"}
Oct 31 16:25:38.901620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:38.901592    4519 generic.go:356] "Generic (PLEG): container finished" podID="0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" containerID="a091982ff84ac56bfe1a60a7cf7d72ec41f0708379833fd89ee48ea64f2f2f2f" exitCode=0
Oct 31 16:25:38.901727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:38.901645    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh" event={"ID":"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32","Type":"ContainerDied","Data":"a091982ff84ac56bfe1a60a7cf7d72ec41f0708379833fd89ee48ea64f2f2f2f"}
Oct 31 16:25:39.019747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:39.019687    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th" podStartSLOduration=2.019668735 podStartE2EDuration="2.019668735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:39.007889913 +0000 UTC m=+1382.707819225" watchObservedRunningTime="2026-10-31 16:25:39.019668735 +0000 UTC m=+1382.719598057"
Oct 31 16:25:39.911263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:39.911234    4519 generic.go:356] "Generic (PLEG): container finished" podID="573809fb-7fa4-459e-8457-127ec7fecd38" containerID="b449aacd8e7c773d1cef8aed072b6078c177915c7984f172bf3e54b1f38cc959" exitCode=0
Oct 31 16:25:39.911673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:39.911322    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th" event={"ID":"573809fb-7fa4-459e-8457-127ec7fecd38","Type":"ContainerDied","Data":"b449aacd8e7c773d1cef8aed072b6078c177915c7984f172bf3e54b1f38cc959"}
Oct 31 16:25:40.078911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.078886    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:40.129689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.129657    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p\" (UniqueName: \"kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p\") pod \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") "
Oct 31 16:25:40.129836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.129742    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca\") pod \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") "
Oct 31 16:25:40.129897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.129881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca\") pod \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\" (UID: \"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32\") "
Oct 31 16:25:40.130341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.130263    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca" pod "0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" (UID: "0920931b-99c7-4ae9-9a9c-c3cef0dc8f32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:40.130664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.130630    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:40.132078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.132044    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca" pod "0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" (UID: "0920931b-99c7-4ae9-9a9c-c3cef0dc8f32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:25:40.132191 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.132098    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p" pod "0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" (UID: "0920931b-99c7-4ae9-9a9c-c3cef0dc8f32"). InnerVolumeSpecName "kube-api-access-ksv5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:40.232054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.232014    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksv5p\" (UniqueName: \"kubernetes.io/projected/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-kube-api-access-ksv5p\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:40.232054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.232055    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:40.690809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.690730    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"]
Oct 31 16:25:40.691541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.691524    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" containerName="revoke-keys"
Oct 31 16:25:40.691587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.691542    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" containerName="revoke-keys"
Oct 31 16:25:40.691677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.691668    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" containerName="revoke-keys"
Oct 31 16:25:40.695713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.695691    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.740589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.740444    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.740748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.740724    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mz9n\" (UniqueName: \"kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.740799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.740783    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.749578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.749535    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"]
Oct 31 16:25:40.842286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.842250    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.842527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.842372    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mz9n\" (UniqueName: \"kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.842527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.842411    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.843120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.843097    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.844717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.844694    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.897069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.897041    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mz9n\" (UniqueName: \"kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n\") pod \"maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:40.921396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.921369    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"
Oct 31 16:25:40.921396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.921384    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh" event={"ID":"0920931b-99c7-4ae9-9a9c-c3cef0dc8f32","Type":"ContainerDied","Data":"4effe0c50b00844a43fc8332d34f2645cf92b9425ec7bd8d44abc5c21b46007f"}
Oct 31 16:25:40.921858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:40.921431    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4effe0c50b00844a43fc8332d34f2645cf92b9425ec7bd8d44abc5c21b46007f"
Oct 31 16:25:41.011868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.011835    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:41.100292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.100263    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:41.147715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.147658    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca\") pod \"573809fb-7fa4-459e-8457-127ec7fecd38\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") "
Oct 31 16:25:41.148062 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.148032    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca" pod "573809fb-7fa4-459e-8457-127ec7fecd38" (UID: "573809fb-7fa4-459e-8457-127ec7fecd38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:41.148211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.148077    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca\") pod \"573809fb-7fa4-459e-8457-127ec7fecd38\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") "
Oct 31 16:25:41.148211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.148113    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s\" (UniqueName: \"kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s\") pod \"573809fb-7fa4-459e-8457-127ec7fecd38\" (UID: \"573809fb-7fa4-459e-8457-127ec7fecd38\") "
Oct 31 16:25:41.148991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.148960    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:41.150648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.150618    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca" pod "573809fb-7fa4-459e-8457-127ec7fecd38" (UID: "573809fb-7fa4-459e-8457-127ec7fecd38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:25:41.150775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.150752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s" pod "573809fb-7fa4-459e-8457-127ec7fecd38" (UID: "573809fb-7fa4-459e-8457-127ec7fecd38"). InnerVolumeSpecName "kube-api-access-s946s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:41.251220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.251135    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573809fb-7fa4-459e-8457-127ec7fecd38-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:41.251220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.251175    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s946s\" (UniqueName: \"kubernetes.io/projected/573809fb-7fa4-459e-8457-127ec7fecd38-kube-api-access-s946s\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:41.411805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.411779    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"]
Oct 31 16:25:41.414754 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:25:41.414727    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ec6033_6199_477d_8590_e3f171dca183.slice/crio-b73e6bb9a7f9639ddd1bfd0e2177f99a3bcc44cff0364e6a7737dd4f338c43c2 WatchSource:0}: Error finding container b73e6bb9a7f9639ddd1bfd0e2177f99a3bcc44cff0364e6a7737dd4f338c43c2: Status 404 returned error can't find the container with id b73e6bb9a7f9639ddd1bfd0e2177f99a3bcc44cff0364e6a7737dd4f338c43c2
Oct 31 16:25:41.931504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.931449    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj" event={"ID":"77ec6033-6199-477d-8590-e3f171dca183","Type":"ContainerStarted","Data":"88ed992513b9116a54a529f234406c197d921a89fe28c73c14e6afed737d5c71"}
Oct 31 16:25:41.931949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.931519    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj" event={"ID":"77ec6033-6199-477d-8590-e3f171dca183","Type":"ContainerStarted","Data":"b73e6bb9a7f9639ddd1bfd0e2177f99a3bcc44cff0364e6a7737dd4f338c43c2"}
Oct 31 16:25:41.933042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.933019    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"
Oct 31 16:25:41.933157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.933022    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th" event={"ID":"573809fb-7fa4-459e-8457-127ec7fecd38","Type":"ContainerDied","Data":"28b6148886dadea1194b9c33f91466ee4a09b2a3dd545e4ffe12920bc33b4acc"}
Oct 31 16:25:41.933157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:41.933133    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28b6148886dadea1194b9c33f91466ee4a09b2a3dd545e4ffe12920bc33b4acc"
Oct 31 16:25:42.049260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:42.049207    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj" podStartSLOduration=2.04919343 podStartE2EDuration="2.04919343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:42.041400967 +0000 UTC m=+1385.741330278" watchObservedRunningTime="2026-10-31 16:25:42.04919343 +0000 UTC m=+1385.749122740"
Oct 31 16:25:42.943129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:42.943097    4519 generic.go:356] "Generic (PLEG): container finished" podID="77ec6033-6199-477d-8590-e3f171dca183" containerID="88ed992513b9116a54a529f234406c197d921a89fe28c73c14e6afed737d5c71" exitCode=0
Oct 31 16:25:42.943556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:42.943167    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj" event={"ID":"77ec6033-6199-477d-8590-e3f171dca183","Type":"ContainerDied","Data":"88ed992513b9116a54a529f234406c197d921a89fe28c73c14e6afed737d5c71"}
Oct 31 16:25:44.135509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.135483    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:44.197608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.197579    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca\") pod \"77ec6033-6199-477d-8590-e3f171dca183\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") "
Oct 31 16:25:44.197812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.197627    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca\") pod \"77ec6033-6199-477d-8590-e3f171dca183\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") "
Oct 31 16:25:44.197933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.197913    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n\" (UniqueName: \"kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n\") pod \"77ec6033-6199-477d-8590-e3f171dca183\" (UID: \"77ec6033-6199-477d-8590-e3f171dca183\") "
Oct 31 16:25:44.198014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.197994    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca" pod "77ec6033-6199-477d-8590-e3f171dca183" (UID: "77ec6033-6199-477d-8590-e3f171dca183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:44.198732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.198684    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ec6033-6199-477d-8590-e3f171dca183-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:44.200176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.200154    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca" pod "77ec6033-6199-477d-8590-e3f171dca183" (UID: "77ec6033-6199-477d-8590-e3f171dca183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:25:44.200737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.200709    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n" pod "77ec6033-6199-477d-8590-e3f171dca183" (UID: "77ec6033-6199-477d-8590-e3f171dca183"). InnerVolumeSpecName "kube-api-access-8mz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:44.300843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.300805    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ec6033-6199-477d-8590-e3f171dca183-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:44.300843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.300840    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mz9n\" (UniqueName: \"kubernetes.io/projected/77ec6033-6199-477d-8590-e3f171dca183-kube-api-access-8mz9n\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:44.963261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.963228    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"
Oct 31 16:25:44.963261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.963256    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj" event={"ID":"77ec6033-6199-477d-8590-e3f171dca183","Type":"ContainerDied","Data":"b73e6bb9a7f9639ddd1bfd0e2177f99a3bcc44cff0364e6a7737dd4f338c43c2"}
Oct 31 16:25:44.963571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:44.963308    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73e6bb9a7f9639ddd1bfd0e2177f99a3bcc44cff0364e6a7737dd4f338c43c2"
Oct 31 16:25:45.069053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:45.069016    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"]
Oct 31 16:25:45.074661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:45.074630    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e0564d-cpg47"]
Oct 31 16:25:46.442606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:25:46.442571    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d173f405-4cd6-43ac-a4b6-ed1337262ef1" path="/var/lib/kubelet/pods/d173f405-4cd6-43ac-a4b6-ed1337262ef1/volumes"
Oct 31 16:25:54.005926 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:25:54.005884    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:26:08.046852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:08.046817    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"]
Oct 31 16:26:08.084011 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:08.083977    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d60d3e-75tdh"]
Oct 31 16:26:08.441088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:08.441009    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3648f5-08e2-49f1-bb84-5c6a1d98c8fd" path="/var/lib/kubelet/pods/af3648f5-08e2-49f1-bb84-5c6a1d98c8fd/volumes"
Oct 31 16:26:15.889674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.889639    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"]
Oct 31 16:26:15.890391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.890375    4519 cpu_manager.go:414] "containerMap: removing container" podUID="77ec6033-6199-477d-8590-e3f171dca183" containerName="revoke-keys"
Oct 31 16:26:15.890391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.890391    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ec6033-6199-477d-8590-e3f171dca183" containerName="revoke-keys"
Oct 31 16:26:15.890538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.890415    4519 cpu_manager.go:414] "containerMap: removing container" podUID="573809fb-7fa4-459e-8457-127ec7fecd38" containerName="revoke-keys"
Oct 31 16:26:15.890538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.890421    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573809fb-7fa4-459e-8457-127ec7fecd38" containerName="revoke-keys"
Oct 31 16:26:15.890608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.890562    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ec6033-6199-477d-8590-e3f171dca183" containerName="revoke-keys"
Oct 31 16:26:15.890608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.890575    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="573809fb-7fa4-459e-8457-127ec7fecd38" containerName="revoke-keys"
Oct 31 16:26:15.895879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.895860    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.924554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924519    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.924673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924574    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.924728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924711    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.924766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924737    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.924799 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924765    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.924971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924952    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.925010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.924988    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9tv\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.925087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.925076    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.925156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.925141    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:15.999297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:15.999263    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"]
Oct 31 16:26:16.026690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.026659    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.026881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.026716    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.026881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.026798    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.026881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.026824    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.026881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.026859    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.027101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.027013    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.027101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.027047    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9tv\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.027203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.027108    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.027203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.027119    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.027307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.027213    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.027928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.027902    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.028249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.028220    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.028922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.028892    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.033682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.033651    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.038727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.034725    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.045825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.045794    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.081651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.081622    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.081843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.081824    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9tv\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.210264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.210184    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:16.648577 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.648552    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"]
Oct 31 16:26:16.650627 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:26:16.650600    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b577f61_a079_4e64_b479_3cd78813f878.slice/crio-4768f49120eba3679f0481555765d8709364933251932538db0759afe91a2ba7 WatchSource:0}: Error finding container 4768f49120eba3679f0481555765d8709364933251932538db0759afe91a2ba7: Status 404 returned error can't find the container with id 4768f49120eba3679f0481555765d8709364933251932538db0759afe91a2ba7
Oct 31 16:26:16.653775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.653737    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:16.653879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.653819    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:16.653879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:16.653863    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:17.326966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:17.326923    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" event={"ID":"6b577f61-a079-4e64-b479-3cd78813f878","Type":"ContainerStarted","Data":"57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f"}
Oct 31 16:26:17.327435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:17.326987    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" event={"ID":"6b577f61-a079-4e64-b479-3cd78813f878","Type":"ContainerStarted","Data":"4768f49120eba3679f0481555765d8709364933251932538db0759afe91a2ba7"}
Oct 31 16:26:17.501851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:17.501796    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podStartSLOduration=2.501780878 podStartE2EDuration="2.501780878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:17.448130839 +0000 UTC m=+1421.148060150" watchObservedRunningTime="2026-10-31 16:26:17.501780878 +0000 UTC m=+1421.201710277"
Oct 31 16:26:18.211036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:18.211006    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:18.212548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:18.212512    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 16:26:18.212678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:18.212568    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 16:26:19.050521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:19.050486    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"]
Oct 31 16:26:19.099354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:19.099320    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-182aaf33-rb67m"]
Oct 31 16:26:19.210677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:19.210637    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 16:26:19.210961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:19.210719    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 16:26:20.211108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:20.211065    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 16:26:20.211660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:20.211147    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 16:26:20.455675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:20.455642    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41857a8e-f80a-43ba-92a7-0079cd8298bd" path="/var/lib/kubelet/pods/41857a8e-f80a-43ba-92a7-0079cd8298bd/volumes"
Oct 31 16:26:21.238480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:21.213867    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 16:26:21.239115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:21.239075    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 16:26:21.245951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:21.245891    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"]
Oct 31 16:26:21.256706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:21.256650    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f27146d0-bgwwl"]
Oct 31 16:26:22.211329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:22.211293    4519 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 31 16:26:22.211541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:22.211348    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 31 16:26:22.447532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:22.447502    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13856c23-ff51-41a7-b150-229f4196297b" path="/var/lib/kubelet/pods/13856c23-ff51-41a7-b150-229f4196297b/volumes"
Oct 31 16:26:22.717850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:22.717820    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"]
Oct 31 16:26:22.750889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:22.750856    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a08562e5-ch6th"]
Oct 31 16:26:23.288064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.288032    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:23.290033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.290010    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:23.291230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.291207    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:23.817914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.817836    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"]
Oct 31 16:26:23.821646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.821605    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-caa664e1-bm2lj"]
Oct 31 16:26:23.892255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.892222    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"]
Oct 31 16:26:23.897630 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.897601    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"]
Oct 31 16:26:23.898080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:23.898046    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy" containerID="cri-o://f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566" gracePeriod=30
Oct 31 16:26:24.073809 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:26:24.073728    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:26:24.443636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:24.443557    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573809fb-7fa4-459e-8457-127ec7fecd38" path="/var/lib/kubelet/pods/573809fb-7fa4-459e-8457-127ec7fecd38/volumes"
Oct 31 16:26:24.444126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:24.444106    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ec6033-6199-477d-8590-e3f171dca183" path="/var/lib/kubelet/pods/77ec6033-6199-477d-8590-e3f171dca183/volumes"
Oct 31 16:26:24.478141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:24.478105    4519 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" secret="" err="secret \"e2e-derive-caa664e1-openshift-default-dockercfg-cbxhc\" not found"
Oct 31 16:26:25.487079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:25.487032    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy" containerID="cri-o://57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f" gracePeriod=30
Oct 31 16:26:29.208602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.208546    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:26:29.323677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.323643    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.323825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.323691    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.323825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.323750    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.323825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.323767    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.323979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.323951    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.324047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324020    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.324164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324147    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.324247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324159    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:29.324247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324184    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.324247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324211    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.324402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324281    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs\") pod \"db282c03-372f-4370-a230-ddc2a1ac3b75\" (UID: \"db282c03-372f-4370-a230-ddc2a1ac3b75\") "
Oct 31 16:26:29.324497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324444    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.324672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324646    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.324765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.324678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.325030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.325007    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.325105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.325031    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.325105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.325041    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.325105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.325050    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.325105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.325058    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db282c03-372f-4370-a230-ddc2a1ac3b75-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.326227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.326202    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.326227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.326208    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.326529 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.326507    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:29.326607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.326544    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5" pod "db282c03-372f-4370-a230-ddc2a1ac3b75" (UID: "db282c03-372f-4370-a230-ddc2a1ac3b75"). InnerVolumeSpecName "kube-api-access-f5gw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.427229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.427195    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5gw5\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-kube-api-access-f5gw5\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.427229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.427228    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.427429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.427240    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.427429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.427249    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db282c03-372f-4370-a230-ddc2a1ac3b75-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.527967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.527930    4519 generic.go:356] "Generic (PLEG): container finished" podID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerID="f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566" exitCode=0
Oct 31 16:26:29.528168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.527986    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"
Oct 31 16:26:29.528168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.528011    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" event={"ID":"db282c03-372f-4370-a230-ddc2a1ac3b75","Type":"ContainerDied","Data":"f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566"}
Oct 31 16:26:29.528168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.528068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk" event={"ID":"db282c03-372f-4370-a230-ddc2a1ac3b75","Type":"ContainerDied","Data":"f837709536aa650ed263772fccd926ba43fe8fd323b476db681078e20b8816c8"}
Oct 31 16:26:29.528168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.528090    4519 scope.go:122] "RemoveContainer" containerID="f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566"
Oct 31 16:26:29.542399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.542376    4519 scope.go:122] "RemoveContainer" containerID="f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566"
Oct 31 16:26:29.542743 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:26:29.542715    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566\": container with ID starting with f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566 not found: ID does not exist" containerID="f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566"
Oct 31 16:26:29.542851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.542755    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566"} err="failed to get container status \"f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566\": rpc error: code = NotFound desc = could not find container \"f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566\": container with ID starting with f2ab52dea846402dba184238780e04df5acd548921a5c82538431881b2e55566 not found: ID does not exist"
Oct 31 16:26:29.687183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.687140    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"]
Oct 31 16:26:29.692423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:29.692399    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-f4vwk"]
Oct 31 16:26:30.443866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.443766    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" path="/var/lib/kubelet/pods/db282c03-372f-4370-a230-ddc2a1ac3b75/volumes"
Oct 31 16:26:30.868209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.868181    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:30.963911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.963885    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964075 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.963940    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964075 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964014    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964159    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964192    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964343    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964404    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964435    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964504    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964565    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964608    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data\") pod \"6b577f61-a079-4e64-b479-3cd78813f878\" (UID: \"6b577f61-a079-4e64-b479-3cd78813f878\") "
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964649    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:30.964816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964690    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:30.965219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.964991    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:30.965499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.965440    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b577f61-a079-4e64-b479-3cd78813f878-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:30.965557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.965509    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:30.965557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.965529    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:30.965557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.965544    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:30.965654 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.965558    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:30.966587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.966559    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:30.966830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.966801    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:30.966921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.966901    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:30.967118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:30.967096    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv" pod "6b577f61-a079-4e64-b479-3cd78813f878" (UID: "6b577f61-a079-4e64-b479-3cd78813f878"). InnerVolumeSpecName "kube-api-access-sv9tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:31.067057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.067018    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9tv\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-kube-api-access-sv9tv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.067057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.067051    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b577f61-a079-4e64-b479-3cd78813f878-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.067057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.067065    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b577f61-a079-4e64-b479-3cd78813f878-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.067292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.067077    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b577f61-a079-4e64-b479-3cd78813f878-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.563487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.562829    4519 generic.go:356] "Generic (PLEG): container finished" podID="6b577f61-a079-4e64-b479-3cd78813f878" containerID="57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f" exitCode=0
Oct 31 16:26:31.563487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.563053    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" event={"ID":"6b577f61-a079-4e64-b479-3cd78813f878","Type":"ContainerDied","Data":"57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f"}
Oct 31 16:26:31.563487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.563106    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc" event={"ID":"6b577f61-a079-4e64-b479-3cd78813f878","Type":"ContainerDied","Data":"4768f49120eba3679f0481555765d8709364933251932538db0759afe91a2ba7"}
Oct 31 16:26:31.563487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.563132    4519 scope.go:122] "RemoveContainer" containerID="57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f"
Oct 31 16:26:31.563487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.563304    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"
Oct 31 16:26:31.588830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.586943    4519 scope.go:122] "RemoveContainer" containerID="57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f"
Oct 31 16:26:31.588830 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:26:31.587266    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f\": container with ID starting with 57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f not found: ID does not exist" containerID="57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f"
Oct 31 16:26:31.588830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.587301    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f"} err="failed to get container status \"57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f\": rpc error: code = NotFound desc = could not find container \"57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f\": container with ID starting with 57300c1a5ebf5d53c25d2e57b436d972ff5cca52cf533897c335fc85d47cd84f not found: ID does not exist"
Oct 31 16:26:31.735500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.735463    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"]
Oct 31 16:26:31.779260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:31.779225    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-pwchc"]
Oct 31 16:26:32.441673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:32.441642    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b577f61-a079-4e64-b479-3cd78813f878" path="/var/lib/kubelet/pods/6b577f61-a079-4e64-b479-3cd78813f878/volumes"
Oct 31 16:26:37.045651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:37.045617    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"]
Oct 31 16:26:37.094386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:37.094347    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-476ef78f-ks9w9"]
Oct 31 16:26:37.829527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:37.829490    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"]
Oct 31 16:26:37.829813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:37.829766    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy" containerID="cri-o://2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b" gracePeriod=30
Oct 31 16:26:38.440328 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:38.440297    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c51aa3-231c-485e-96d5-943173aecd5a" path="/var/lib/kubelet/pods/91c51aa3-231c-485e-96d5-943173aecd5a/volumes"
Oct 31 16:26:43.129490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.129443    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:26:43.267323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267293    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267403    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267506    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267591    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267654    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267688    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267743    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.267884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267813    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:43.267884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267852    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:43.267987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267955    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.268041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.267991    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq\") pod \"059db11a-da47-4d8f-b774-236d0b42e3ed\" (UID: \"059db11a-da47-4d8f-b774-236d0b42e3ed\") "
Oct 31 16:26:43.268227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268183    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:43.268227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268207    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:43.268403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268251    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:43.268855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268830    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.268957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268859    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.268957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268874    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.268957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268887    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.268957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.268902    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059db11a-da47-4d8f-b774-236d0b42e3ed-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.269850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.269825    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:43.269960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.269941    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:43.270230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.270204    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:43.270629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.270605    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq" pod "059db11a-da47-4d8f-b774-236d0b42e3ed" (UID: "059db11a-da47-4d8f-b774-236d0b42e3ed"). InnerVolumeSpecName "kube-api-access-xdmdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:43.370281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.370202    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.370281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.370231    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdmdq\" (UniqueName: \"kubernetes.io/projected/059db11a-da47-4d8f-b774-236d0b42e3ed-kube-api-access-xdmdq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.370281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.370242    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.370281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.370252    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059db11a-da47-4d8f-b774-236d0b42e3ed-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:43.592229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.592196    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:26:43.690090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.690000    4519 generic.go:356] "Generic (PLEG): container finished" podID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerID="2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b" exitCode=0
Oct 31 16:26:43.690090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.690076    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"
Oct 31 16:26:43.690255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.690079    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" event={"ID":"059db11a-da47-4d8f-b774-236d0b42e3ed","Type":"ContainerDied","Data":"2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b"}
Oct 31 16:26:43.690255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.690212    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn" event={"ID":"059db11a-da47-4d8f-b774-236d0b42e3ed","Type":"ContainerDied","Data":"8be2a370bd736f50839da746d55a7ce77524176c5af4c7be45cda0093334a684"}
Oct 31 16:26:43.690255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.690237    4519 scope.go:122] "RemoveContainer" containerID="2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b"
Oct 31 16:26:43.711182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.711157    4519 scope.go:122] "RemoveContainer" containerID="2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b"
Oct 31 16:26:43.711533 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:26:43.711506    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b\": container with ID starting with 2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b not found: ID does not exist" containerID="2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b"
Oct 31 16:26:43.711598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.711544    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b"} err="failed to get container status \"2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b\": rpc error: code = NotFound desc = could not find container \"2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b\": container with ID starting with 2c3d6046c950fb4b91169e3c72892d9d06d3642f93a4e7af9090ad6a450acd1b not found: ID does not exist"
Oct 31 16:26:43.854582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.854539    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"]
Oct 31 16:26:43.859506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:43.859481    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-2lprn"]
Oct 31 16:26:44.442294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:44.442258    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" path="/var/lib/kubelet/pods/059db11a-da47-4d8f-b774-236d0b42e3ed/volumes"
Oct 31 16:26:47.390478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:47.389581    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:47.419594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:47.419539    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:47.466751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:47.466706    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:47.561634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:47.561590    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:26:47.561955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:47.561905    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:53.048605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:53.048569    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"]
Oct 31 16:26:53.095753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:53.095705    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b3ee55-h5wn2"]
Oct 31 16:26:54.004796 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:26:54.004760    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:26:54.441679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:26:54.441599    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8eaac28-479b-4cde-8c80-3977515ca30a" path="/var/lib/kubelet/pods/c8eaac28-479b-4cde-8c80-3977515ca30a/volumes"
Oct 31 16:27:03.047907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:03.047868    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"]
Oct 31 16:27:03.095636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:03.095599    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a06b5dca-ptmd8"]
Oct 31 16:27:04.442491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:04.442438    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0162124-ae0f-4f68-9410-a67166726c76" path="/var/lib/kubelet/pods/b0162124-ae0f-4f68-9410-a67166726c76/volumes"
Oct 31 16:27:08.203905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.203861    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"]
Oct 31 16:27:08.204821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.204797    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy"
Oct 31 16:27:08.204821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.204820    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy"
Oct 31 16:27:08.205025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.204862    4519 cpu_manager.go:414] "containerMap: removing container" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy"
Oct 31 16:27:08.205025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.204870    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy"
Oct 31 16:27:08.205025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.204915    4519 cpu_manager.go:414] "containerMap: removing container" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy"
Oct 31 16:27:08.205025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.204923    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy"
Oct 31 16:27:08.205246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.205092    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b577f61-a079-4e64-b479-3cd78813f878" containerName="istio-proxy"
Oct 31 16:27:08.205246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.205113    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="db282c03-372f-4370-a230-ddc2a1ac3b75" containerName="istio-proxy"
Oct 31 16:27:08.205246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.205132    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="059db11a-da47-4d8f-b774-236d0b42e3ed" containerName="istio-proxy"
Oct 31 16:27:08.209387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.209366    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.398487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.394205    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"]
Oct 31 16:27:08.398487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.394948    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.398487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.395019    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.398487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.395086    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.402652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.399867    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-d04acd-openshift-default-dockercfg-pplxx\""
Oct 31 16:27:08.410572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.410541    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.410993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.410965    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9l5h\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.411213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.411197    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.411349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.411336    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.411569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.411553    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.411821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.411808    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514067    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514167    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514218    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514294    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514370    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l5h\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514433    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514500    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.514751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.514734    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.515523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.515499    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.516603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.516551    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.517031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.517003    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.517131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.517061    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.517269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.517249    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.519695 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.519665    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.519868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.519850    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.568639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.568612    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.570020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.569988    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9l5h\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h\") pod \"e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") " pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:08.823817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:08.823730    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:09.259986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:09.259947    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"]
Oct 31 16:27:09.302662 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:09.302631    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f8058a_6c74_450b_8481_0e5900e34651.slice/crio-8ff16e7a987ebdc91f7bbd181dfd78b778cc10b0b9db6e1d111f5e840ce9ee40 WatchSource:0}: Error finding container 8ff16e7a987ebdc91f7bbd181dfd78b778cc10b0b9db6e1d111f5e840ce9ee40: Status 404 returned error can't find the container with id 8ff16e7a987ebdc91f7bbd181dfd78b778cc10b0b9db6e1d111f5e840ce9ee40
Oct 31 16:27:09.305296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:09.305262    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:09.305385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:09.305330    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:09.305385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:09.305363    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:10.080047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:10.080002    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" event={"ID":"23f8058a-6c74-450b-8481-0e5900e34651","Type":"ContainerStarted","Data":"b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"}
Oct 31 16:27:10.080239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:10.080071    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" event={"ID":"23f8058a-6c74-450b-8481-0e5900e34651","Type":"ContainerStarted","Data":"8ff16e7a987ebdc91f7bbd181dfd78b778cc10b0b9db6e1d111f5e840ce9ee40"}
Oct 31 16:27:10.824674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:10.824647    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:10.866511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:10.865655    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:11.052215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:11.052143    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podStartSLOduration=3.052124849 podStartE2EDuration="3.052124849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:10.284908163 +0000 UTC m=+1473.984837474" watchObservedRunningTime="2026-10-31 16:27:11.052124849 +0000 UTC m=+1474.752054161"
Oct 31 16:27:11.089721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:11.089641    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:11.090678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:11.090658    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:27:18.380959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.380865    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"]
Oct 31 16:27:18.386818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.386788    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.445423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.445388    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"]
Oct 31 16:27:18.535578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.535542    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-d04acd\""
Oct 31 16:27:18.542263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.536079    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-d04acd-dockercfg-n7t5p\""
Oct 31 16:27:18.542534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.541327    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.542735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.542716    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj56h\" (UniqueName: \"kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h\") pod \"payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.582661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.582623    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"]
Oct 31 16:27:18.590351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.590323    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.649126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.649077    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj56h\" (UniqueName: \"kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h\") pod \"payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.649940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.649914    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.651835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.651796    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.663223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.663191    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"]
Oct 31 16:27:18.706944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.706124    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj56h\" (UniqueName: \"kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h\") pod \"payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.718503 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.715300    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:18.751807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.751759    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.751991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.751953    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8kfd\" (UniqueName: \"kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd\") pod \"payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.857562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.853972    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.857562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.854163    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8kfd\" (UniqueName: \"kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd\") pod \"payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.857562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.855586    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.912128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.912044    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8kfd\" (UniqueName: \"kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd\") pod \"payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:18.913292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.913268    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"]
Oct 31 16:27:18.920414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.920390    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:18.945021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.944986    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:19.000625 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:18.976099    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a7907d70-openshift-default-dockercfg-2jwjv\""
Oct 31 16:27:19.050560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.050393    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"]
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.095680    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.095751    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.095853    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lp85\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.095909    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.096001    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.096040    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.096128    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.096194    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.096968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.096295    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.161399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.156529    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"]
Oct 31 16:27:19.171519 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:19.168775    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b59a33_3dac_49d9_9d72_b932840a530c.slice/crio-6e73522743d9af01a16374014f0e93295940af3f020d06f06c14f770772eba01 WatchSource:0}: Error finding container 6e73522743d9af01a16374014f0e93295940af3f020d06f06c14f770772eba01: Status 404 returned error can't find the container with id 6e73522743d9af01a16374014f0e93295940af3f020d06f06c14f770772eba01
Oct 31 16:27:19.197873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.197820    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lp85\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.198670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.198641    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.199629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.199606    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.200212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.200193    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.200473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.200441    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.202553    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.202672    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.199475    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.203042    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.203295    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.203819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.204664    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.204834    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.209225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.205045    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.214350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.214008    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.229243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.229158    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.236096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.236053    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2" event={"ID":"20b59a33-3dac-49d9-9d72-b932840a530c","Type":"ContainerStarted","Data":"6e73522743d9af01a16374014f0e93295940af3f020d06f06c14f770772eba01"}
Oct 31 16:27:19.254444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.253804    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lp85\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.263695 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.263530    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token\") pod \"e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") " pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.324703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.324665    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:19.473864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.473828    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"]
Oct 31 16:27:19.475697 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:19.475664    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3a725f_5a5f_4bb7_9843_f0374fcb1b23.slice/crio-43d45be75d3ed431d4b79dd8d7e06bc905abd9ea59fd6f3407cf72300ceee383 WatchSource:0}: Error finding container 43d45be75d3ed431d4b79dd8d7e06bc905abd9ea59fd6f3407cf72300ceee383: Status 404 returned error can't find the container with id 43d45be75d3ed431d4b79dd8d7e06bc905abd9ea59fd6f3407cf72300ceee383
Oct 31 16:27:19.732144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.732116    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"]
Oct 31 16:27:19.733236 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:19.733210    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821daccc_0a90_4511_9c5e_613589b2681c.slice/crio-add3171b3e48c4fa763427c0bee0846f59ea6dd192b8b2b6f27db496b398602a WatchSource:0}: Error finding container add3171b3e48c4fa763427c0bee0846f59ea6dd192b8b2b6f27db496b398602a: Status 404 returned error can't find the container with id add3171b3e48c4fa763427c0bee0846f59ea6dd192b8b2b6f27db496b398602a
Oct 31 16:27:19.736168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.736134    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:19.736249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.736203    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:19.736249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:19.736234    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:20.052129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.052043    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"]
Oct 31 16:27:20.057086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.057054    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56adc755-rmvht"]
Oct 31 16:27:20.246444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.246408    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" event={"ID":"821daccc-0a90-4511-9c5e-613589b2681c","Type":"ContainerStarted","Data":"2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"}
Oct 31 16:27:20.246659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.246491    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" event={"ID":"821daccc-0a90-4511-9c5e-613589b2681c","Type":"ContainerStarted","Data":"add3171b3e48c4fa763427c0bee0846f59ea6dd192b8b2b6f27db496b398602a"}
Oct 31 16:27:20.248253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.248223    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2" event={"ID":"20b59a33-3dac-49d9-9d72-b932840a530c","Type":"ContainerStarted","Data":"025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074"}
Oct 31 16:27:20.248403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.248314    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:20.250029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.250003    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2" event={"ID":"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23","Type":"ContainerStarted","Data":"c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2"}
Oct 31 16:27:20.250135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.250048    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2" event={"ID":"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23","Type":"ContainerStarted","Data":"43d45be75d3ed431d4b79dd8d7e06bc905abd9ea59fd6f3407cf72300ceee383"}
Oct 31 16:27:20.250135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.250086    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:20.325036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.324954    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:20.331672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.331646    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:20.361221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.361165    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podStartSLOduration=2.361153335 podStartE2EDuration="2.361153335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:20.358047423 +0000 UTC m=+1484.057976736" watchObservedRunningTime="2026-10-31 16:27:20.361153335 +0000 UTC m=+1484.061082645"
Oct 31 16:27:20.441548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.441507    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a645cb-02bf-47c5-8b80-b4882c50d9a2" path="/var/lib/kubelet/pods/87a645cb-02bf-47c5-8b80-b4882c50d9a2/volumes"
Oct 31 16:27:20.583984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.583817    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2" podStartSLOduration=2.5837961099999998 podStartE2EDuration="2.58379611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:20.577069999 +0000 UTC m=+1484.276999310" watchObservedRunningTime="2026-10-31 16:27:20.58379611 +0000 UTC m=+1484.283725422"
Oct 31 16:27:20.687169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:20.687117    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2" podStartSLOduration=2.687101561 podStartE2EDuration="2.687101561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:20.684687457 +0000 UTC m=+1484.384616768" watchObservedRunningTime="2026-10-31 16:27:20.687101561 +0000 UTC m=+1484.387030871"
Oct 31 16:27:21.048254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:21.048213    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"]
Oct 31 16:27:21.053222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:21.053194    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bf0f66-ctbjv"]
Oct 31 16:27:21.259562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:21.259524    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:21.260324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:21.260294    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:27:22.442486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:22.442439    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f6b22a-0d41-4281-a055-d6b1ab7ff2b0" path="/var/lib/kubelet/pods/09f6b22a-0d41-4281-a055-d6b1ab7ff2b0/volumes"
Oct 31 16:27:23.092385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:23.092349    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"]
Oct 31 16:27:23.097665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:23.097638    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fb4b73-sc5kd"]
Oct 31 16:27:24.006959 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:27:24.006924    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:27:24.443059 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:24.442979    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8dbf549-6b29-49f0-9769-36c73c81e0a8" path="/var/lib/kubelet/pods/e8dbf549-6b29-49f0-9769-36c73c81e0a8/volumes"
Oct 31 16:27:26.680883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.680852    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"]
Oct 31 16:27:26.685510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.685489    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:26.746812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.746780    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"]
Oct 31 16:27:26.785862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.785006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2x8k\" (UniqueName: \"kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k\") pod \"payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:26.785862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.785166    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:26.788516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.787167    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a7907d70\""
Oct 31 16:27:26.788516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.787335    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a7907d70-dockercfg-vr5gc\""
Oct 31 16:27:26.891478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.886574    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:26.891478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.886968    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2x8k\" (UniqueName: \"kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k\") pod \"payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:26.891478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.887894    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:26.941484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:26.941394    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2x8k\" (UniqueName: \"kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k\") pod \"payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:27.003248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:27.003190    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:27.401042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:27.400956    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"]
Oct 31 16:27:27.403839 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:27.403810    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b5bcd9_dfe5_4f81_89d6_d33bcdda3564.slice/crio-f73721841a72720cd58c92d81a80c2e7ffe6afec543247818661dc8dfc09f9be WatchSource:0}: Error finding container f73721841a72720cd58c92d81a80c2e7ffe6afec543247818661dc8dfc09f9be: Status 404 returned error can't find the container with id f73721841a72720cd58c92d81a80c2e7ffe6afec543247818661dc8dfc09f9be
Oct 31 16:27:28.332086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:28.332053    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n" event={"ID":"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564","Type":"ContainerStarted","Data":"ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29"}
Oct 31 16:27:28.332513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:28.332102    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n" event={"ID":"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564","Type":"ContainerStarted","Data":"f73721841a72720cd58c92d81a80c2e7ffe6afec543247818661dc8dfc09f9be"}
Oct 31 16:27:28.332513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:28.332136    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:28.444948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:28.444898    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n" podStartSLOduration=2.4448838520000002 podStartE2EDuration="2.444883852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:28.440715157 +0000 UTC m=+1492.140644470" watchObservedRunningTime="2026-10-31 16:27:28.444883852 +0000 UTC m=+1492.144813165"
Oct 31 16:27:31.262131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:31.262101    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:27:31.262568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:31.262176    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:27:33.970519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:33.970486    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"]
Oct 31 16:27:33.975334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:33.975313    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.027043 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.026437    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:27:34.027418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.027374    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:27:34.027636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.026632    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:27:34.063140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.039695    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"]
Oct 31 16:27:34.063140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.051377    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.063140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.051479    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.063140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.051612    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tb5\" (UniqueName: \"kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.071124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.071099    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:27:34.153563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.153527    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.153729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.153631    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.153773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.153743    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tb5\" (UniqueName: \"kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.154284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.154263    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.156045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.156019    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.169096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.166808    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"]
Oct 31 16:27:34.179484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.179435    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269073    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269171    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269211    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269245    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269298    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269356    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269400    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269437    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.269661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.269562    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrlt\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.298570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.296481    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"]
Oct 31 16:27:34.298570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.297183    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tb5\" (UniqueName: \"kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5\") pod \"maas-api-revoke-sub-102e1ef5-lwx7k\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.331476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.328136    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-dc67dd-openshift-default-dockercfg-nfsx5\""
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.371892    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.371983    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372027    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372058    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372120    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372176    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372218    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372257    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.372360    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrlt\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.373435    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.373692    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.374478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.374159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.388663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.378113    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.388663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.378832    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.415516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.414677    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.431476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.430123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.441556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.434360    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrlt\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.441556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.440756    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token\") pod \"e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") " pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.589506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.589391    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:34.635507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.635476    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:34.983761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:34.983727    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"]
Oct 31 16:27:34.987377 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:34.987349    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4660b8d3_720d_4c97_8ff9_e1325a82ffb8.slice/crio-8111096809a82acf936b5ae44cd565944a39be73d45449a66276ad37d7187b1f WatchSource:0}: Error finding container 8111096809a82acf936b5ae44cd565944a39be73d45449a66276ad37d7187b1f: Status 404 returned error can't find the container with id 8111096809a82acf936b5ae44cd565944a39be73d45449a66276ad37d7187b1f
Oct 31 16:27:35.098913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.098885    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"]
Oct 31 16:27:35.168688 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:35.168651    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d9cc78_f6fb_4031_a540_adc0d2e4fadd.slice/crio-97ea85423e427a349a512fa5bf991c3648c95e5af67e08d9d3a3e3c7ebf05898 WatchSource:0}: Error finding container 97ea85423e427a349a512fa5bf991c3648c95e5af67e08d9d3a3e3c7ebf05898: Status 404 returned error can't find the container with id 97ea85423e427a349a512fa5bf991c3648c95e5af67e08d9d3a3e3c7ebf05898
Oct 31 16:27:35.171645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.171609    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:35.171738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.171682    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:35.171738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.171709    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:27:35.433248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.433215    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k" event={"ID":"4660b8d3-720d-4c97-8ff9-e1325a82ffb8","Type":"ContainerStarted","Data":"9ef63a9bbb1823188647516416fcab1a09216f7ae99f9efe3b8dd4a39b34309e"}
Oct 31 16:27:35.433445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.433266    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k" event={"ID":"4660b8d3-720d-4c97-8ff9-e1325a82ffb8","Type":"ContainerStarted","Data":"8111096809a82acf936b5ae44cd565944a39be73d45449a66276ad37d7187b1f"}
Oct 31 16:27:35.435361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.435335    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" event={"ID":"40d9cc78-f6fb-4031-a540-adc0d2e4fadd","Type":"ContainerStarted","Data":"76b24565c90e9a45bbf3017326f917216b3f7a4a5f6b507013e55d3053b298cc"}
Oct 31 16:27:35.435498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.435379    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" event={"ID":"40d9cc78-f6fb-4031-a540-adc0d2e4fadd","Type":"ContainerStarted","Data":"97ea85423e427a349a512fa5bf991c3648c95e5af67e08d9d3a3e3c7ebf05898"}
Oct 31 16:27:35.542669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.542559    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k" podStartSLOduration=2.542545108 podStartE2EDuration="2.542545108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:35.539873758 +0000 UTC m=+1499.239803069" watchObservedRunningTime="2026-10-31 16:27:35.542545108 +0000 UTC m=+1499.242474419"
Oct 31 16:27:35.636300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.636271    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:35.655188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:35.655127    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podStartSLOduration=1.655111126 podStartE2EDuration="1.655111126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:35.649386732 +0000 UTC m=+1499.349316044" watchObservedRunningTime="2026-10-31 16:27:35.655111126 +0000 UTC m=+1499.355040437"
Oct 31 16:27:36.456514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:36.456395    4519 generic.go:356] "Generic (PLEG): container finished" podID="4660b8d3-720d-4c97-8ff9-e1325a82ffb8" containerID="9ef63a9bbb1823188647516416fcab1a09216f7ae99f9efe3b8dd4a39b34309e" exitCode=0
Oct 31 16:27:36.456956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:36.456759    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k" event={"ID":"4660b8d3-720d-4c97-8ff9-e1325a82ffb8","Type":"ContainerDied","Data":"9ef63a9bbb1823188647516416fcab1a09216f7ae99f9efe3b8dd4a39b34309e"}
Oct 31 16:27:36.642819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:36.642792    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:37.045071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.045035    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"]
Oct 31 16:27:37.049982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.049955    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4d54e6-8h86c"]
Oct 31 16:27:37.270425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.270393    4519 scope.go:122] "RemoveContainer" containerID="1612e7aa16a029fa7758096538718a877f9755588672a9f411eb18748b3a277e"
Oct 31 16:27:37.281475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.281432    4519 scope.go:122] "RemoveContainer" containerID="44e9a39112259acfc3ee83ba75874fec2d0d0a1eba2b706a619b33d2863a77d4"
Oct 31 16:27:37.291573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.291543    4519 scope.go:122] "RemoveContainer" containerID="e8c78be1c5791c898c2953bbb32b3205d0bab2b6a2ac5ac5898c6de914128095"
Oct 31 16:27:37.303741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.303708    4519 scope.go:122] "RemoveContainer" containerID="8f8a8c37ad4a6cdd134289def9e87f058f7dc4d3b194210d10fc14aed780f23f"
Oct 31 16:27:37.465155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.465114    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:37.466209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.466189    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:27:37.679244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.679220    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:37.747327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.747296    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5\" (UniqueName: \"kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5\") pod \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") "
Oct 31 16:27:37.747522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.747474    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca\") pod \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") "
Oct 31 16:27:37.747634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.747621    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca\") pod \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\" (UID: \"4660b8d3-720d-4c97-8ff9-e1325a82ffb8\") "
Oct 31 16:27:37.747837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.747813    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca" pod "4660b8d3-720d-4c97-8ff9-e1325a82ffb8" (UID: "4660b8d3-720d-4c97-8ff9-e1325a82ffb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:27:37.748219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.748202    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:27:37.749505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.749442    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5" pod "4660b8d3-720d-4c97-8ff9-e1325a82ffb8" (UID: "4660b8d3-720d-4c97-8ff9-e1325a82ffb8"). InnerVolumeSpecName "kube-api-access-h9tb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:27:37.749592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.749558    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca" pod "4660b8d3-720d-4c97-8ff9-e1325a82ffb8" (UID: "4660b8d3-720d-4c97-8ff9-e1325a82ffb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:27:37.850246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.850206    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:27:37.850246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:37.850235    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9tb5\" (UniqueName: \"kubernetes.io/projected/4660b8d3-720d-4c97-8ff9-e1325a82ffb8-kube-api-access-h9tb5\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:27:38.451221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:38.451189    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d0fa60-1708-423a-9e9c-818b8930977c" path="/var/lib/kubelet/pods/71d0fa60-1708-423a-9e9c-818b8930977c/volumes"
Oct 31 16:27:38.496648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:38.484697    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"
Oct 31 16:27:38.496648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:38.484748    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k" event={"ID":"4660b8d3-720d-4c97-8ff9-e1325a82ffb8","Type":"ContainerDied","Data":"8111096809a82acf936b5ae44cd565944a39be73d45449a66276ad37d7187b1f"}
Oct 31 16:27:38.496648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:38.484970    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8111096809a82acf936b5ae44cd565944a39be73d45449a66276ad37d7187b1f"
Oct 31 16:27:39.343957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:39.343926    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:27:42.184886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.184849    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"]
Oct 31 16:27:42.185867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.185841    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4660b8d3-720d-4c97-8ff9-e1325a82ffb8" containerName="revoke-keys"
Oct 31 16:27:42.185867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.185867    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4660b8d3-720d-4c97-8ff9-e1325a82ffb8" containerName="revoke-keys"
Oct 31 16:27:42.186097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.186055    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4660b8d3-720d-4c97-8ff9-e1325a82ffb8" containerName="revoke-keys"
Oct 31 16:27:42.191375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.191353    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.239405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.238604    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.239405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.238790    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv668\" (UniqueName: \"kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668\") pod \"payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.243553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.243521    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"]
Oct 31 16:27:42.280833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.280800    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dc67dd\""
Oct 31 16:27:42.281040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.280841    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dc67dd-dockercfg-d554n\""
Oct 31 16:27:42.343825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.343764    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.344101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.343960    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv668\" (UniqueName: \"kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668\") pod \"payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.346609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.346582    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.402659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.402626    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv668\" (UniqueName: \"kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668\") pod \"payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.510599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.510562    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:42.918629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:42.918418    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"]
Oct 31 16:27:42.921784 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:27:42.921759    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bdbf93_f01c_4801_8fec_71e606450eaa.slice/crio-a7941d4db693814afed7d90d8b2c9e673c8c5f1325df39060905a45cbf176edb WatchSource:0}: Error finding container a7941d4db693814afed7d90d8b2c9e673c8c5f1325df39060905a45cbf176edb: Status 404 returned error can't find the container with id a7941d4db693814afed7d90d8b2c9e673c8c5f1325df39060905a45cbf176edb
Oct 31 16:27:43.546187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:43.546155    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl" event={"ID":"52bdbf93-f01c-4801-8fec-71e606450eaa","Type":"ContainerStarted","Data":"6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e"}
Oct 31 16:27:43.546685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:43.546206    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl" event={"ID":"52bdbf93-f01c-4801-8fec-71e606450eaa","Type":"ContainerStarted","Data":"a7941d4db693814afed7d90d8b2c9e673c8c5f1325df39060905a45cbf176edb"}
Oct 31 16:27:43.546685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:43.546274    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:43.662112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:43.662042    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl" podStartSLOduration=1.662023741 podStartE2EDuration="1.662023741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:43.65432864 +0000 UTC m=+1507.354257952" watchObservedRunningTime="2026-10-31 16:27:43.662023741 +0000 UTC m=+1507.361953056"
Oct 31 16:27:47.047646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:47.047604    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"]
Oct 31 16:27:47.094794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:47.094758    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"]
Oct 31 16:27:47.100254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:47.100222    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4519e810-7z7js"]
Oct 31 16:27:47.105141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:47.105114    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04b8bdf-xqkns"]
Oct 31 16:27:48.444174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:48.444135    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627bd569-d8c2-4590-a2ba-5721ab406251" path="/var/lib/kubelet/pods/627bd569-d8c2-4590-a2ba-5721ab406251/volumes"
Oct 31 16:27:48.444801 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:48.444783    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a50218e-f83c-4f38-9c36-bf5f9a61a791" path="/var/lib/kubelet/pods/9a50218e-f83c-4f38-9c36-bf5f9a61a791/volumes"
Oct 31 16:27:51.058468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:51.058424    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"]
Oct 31 16:27:51.105535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:51.105503    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"]
Oct 31 16:27:51.111163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:51.111133    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def6daf6-8jj8l"]
Oct 31 16:27:51.116145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:51.116120    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecde7be7-6mw7z"]
Oct 31 16:27:51.564624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:51.564589    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:27:52.046219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:52.046181    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"]
Oct 31 16:27:52.050907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:52.050879    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef8190-sfj8g"]
Oct 31 16:27:52.441703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:52.441626    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef8b50f-073d-4e5a-933d-2321cfd82d98" path="/var/lib/kubelet/pods/1ef8b50f-073d-4e5a-933d-2321cfd82d98/volumes"
Oct 31 16:27:52.442203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:52.442187    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9467039d-55c5-4574-91d9-cd1023b29d0d" path="/var/lib/kubelet/pods/9467039d-55c5-4574-91d9-cd1023b29d0d/volumes"
Oct 31 16:27:52.442683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:52.442667    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bff5c0-a483-495c-b4c0-05159a8f4e73" path="/var/lib/kubelet/pods/a4bff5c0-a483-495c-b4c0-05159a8f4e73/volumes"
Oct 31 16:27:54.006590 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:27:54.006554    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:27:54.558465 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:54.558434    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:27:55.044339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:55.044305    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"]
Oct 31 16:27:55.049340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:55.049311    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18571a62-jgzzd"]
Oct 31 16:27:56.441413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:56.441380    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa8ea29-d370-422f-869e-98fb3df77078" path="/var/lib/kubelet/pods/4fa8ea29-d370-422f-869e-98fb3df77078/volumes"
Oct 31 16:27:58.046020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:58.045985    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"]
Oct 31 16:27:58.093805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:58.093772    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e66d246-q7qmq"]
Oct 31 16:27:58.444438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:27:58.444360    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d5cddc-a102-4ba4-a58c-8b1d531865ef" path="/var/lib/kubelet/pods/07d5cddc-a102-4ba4-a58c-8b1d531865ef/volumes"
Oct 31 16:28:00.045747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:00.045715    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"]
Oct 31 16:28:00.092670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:00.092634    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1641897-sqtb8"]
Oct 31 16:28:00.441363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:00.441288    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40cdf4f5-4d95-4819-bf83-73ab1a700db7" path="/var/lib/kubelet/pods/40cdf4f5-4d95-4819-bf83-73ab1a700db7/volumes"
Oct 31 16:28:02.044669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:02.044634    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"]
Oct 31 16:28:02.091966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:02.091934    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa28b1a-bd8ll"]
Oct 31 16:28:02.444183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:02.444098    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c547c7-c8f7-40b2-9961-e2d14cfbe641" path="/var/lib/kubelet/pods/95c547c7-c8f7-40b2-9961-e2d14cfbe641/volumes"
Oct 31 16:28:08.045204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:08.045166    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"]
Oct 31 16:28:08.091865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:08.091836    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c8b07-qlw6b"]
Oct 31 16:28:08.441764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:08.441680    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b99b10c-0d1c-4c52-af9d-c00bb23475ff" path="/var/lib/kubelet/pods/8b99b10c-0d1c-4c52-af9d-c00bb23475ff/volumes"
Oct 31 16:28:10.097792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.097760    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"]
Oct 31 16:28:10.111512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.111162    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.153932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.153901    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"]
Oct 31 16:28:10.195755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.195725    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:28:10.195755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.195725    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:28:10.196019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.195774    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:28:10.208816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.208788    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:28:10.237531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.237439    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.237732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.237714    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w628b\" (UniqueName: \"kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.237777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.237745    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.339376 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.339340    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w628b\" (UniqueName: \"kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.339376 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.339379    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.339795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.339655    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.340258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.340240    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.341758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.341732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.393268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.393194    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w628b\" (UniqueName: \"kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b\") pod \"maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.426690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.426664    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:10.821699 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:10.821659    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"]
Oct 31 16:28:10.827175 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:28:10.827144    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b1ff9c_1db2_451e_88f0_adc1f3daf14f.slice/crio-65532c26f11c194091cf548eac93dbcd06fdf2265f892c42adb9d3f88f43cf24 WatchSource:0}: Error finding container 65532c26f11c194091cf548eac93dbcd06fdf2265f892c42adb9d3f88f43cf24: Status 404 returned error can't find the container with id 65532c26f11c194091cf548eac93dbcd06fdf2265f892c42adb9d3f88f43cf24
Oct 31 16:28:11.841368 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:11.841330    4519 generic.go:356] "Generic (PLEG): container finished" podID="c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" containerID="44a03de651a0c9411ac95bcc42717a86e1066be8673e5658f6ae858c49232f6b" exitCode=0
Oct 31 16:28:11.841768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:11.841369    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft" event={"ID":"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f","Type":"ContainerDied","Data":"44a03de651a0c9411ac95bcc42717a86e1066be8673e5658f6ae858c49232f6b"}
Oct 31 16:28:11.841768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:11.841427    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft" event={"ID":"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f","Type":"ContainerStarted","Data":"65532c26f11c194091cf548eac93dbcd06fdf2265f892c42adb9d3f88f43cf24"}
Oct 31 16:28:13.130976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.130894    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:13.145634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.145604    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"]
Oct 31 16:28:13.178477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.173541    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f3d3f21-js5hk"]
Oct 31 16:28:13.193255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.193222    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca\") pod \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") "
Oct 31 16:28:13.193643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.193618    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca\") pod \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") "
Oct 31 16:28:13.193849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.193834    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b\" (UniqueName: \"kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b\") pod \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\" (UID: \"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f\") "
Oct 31 16:28:13.195784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.195761    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca" pod "c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" (UID: "c8b1ff9c-1db2-451e-88f0-adc1f3daf14f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:28:13.197697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.197659    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b" pod "c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" (UID: "c8b1ff9c-1db2-451e-88f0-adc1f3daf14f"). InnerVolumeSpecName "kube-api-access-w628b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:28:13.216231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.214543    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca" pod "c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" (UID: "c8b1ff9c-1db2-451e-88f0-adc1f3daf14f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:28:13.296131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.296096    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:13.296131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.296129    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:13.296373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.296145    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w628b\" (UniqueName: \"kubernetes.io/projected/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f-kube-api-access-w628b\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:13.861034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.861007    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"
Oct 31 16:28:13.861239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.861005    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft" event={"ID":"c8b1ff9c-1db2-451e-88f0-adc1f3daf14f","Type":"ContainerDied","Data":"65532c26f11c194091cf548eac93dbcd06fdf2265f892c42adb9d3f88f43cf24"}
Oct 31 16:28:13.861239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:13.861149    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65532c26f11c194091cf548eac93dbcd06fdf2265f892c42adb9d3f88f43cf24"
Oct 31 16:28:14.442196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:14.442164    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28681382-119d-4b1c-ba27-bb0196abf0ac" path="/var/lib/kubelet/pods/28681382-119d-4b1c-ba27-bb0196abf0ac/volumes"
Oct 31 16:28:16.044398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:16.044366    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"]
Oct 31 16:28:16.091242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:16.091203    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df8775-48ldc"]
Oct 31 16:28:16.442491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:16.442389    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27a37e3-fd04-4e4b-acbf-297df257d4c9" path="/var/lib/kubelet/pods/b27a37e3-fd04-4e4b-acbf-297df257d4c9/volumes"
Oct 31 16:28:17.935495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:17.935448    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"]
Oct 31 16:28:17.936235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:17.936220    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" containerName="revoke-keys"
Oct 31 16:28:17.936235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:17.936236    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" containerName="revoke-keys"
Oct 31 16:28:17.936363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:17.936353    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" containerName="revoke-keys"
Oct 31 16:28:17.941653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:17.941634    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:17.992602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:17.992572    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"]
Oct 31 16:28:18.033446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.033399    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:28:18.033446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.033418    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:28:18.033446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.033426    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:28:18.038575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.038546    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:28:18.080511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.080450    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.080656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.080618    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5msrm\" (UniqueName: \"kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.080703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.080672    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.182323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.182286    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.182538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.182370    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5msrm\" (UniqueName: \"kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.182538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.182412    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.183087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.183066    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.184736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.184713    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.236001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.235970    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5msrm\" (UniqueName: \"kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm\") pod \"maas-api-revoke-keys-e2e-ait-ce4664d0-tr644\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.256923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.256888    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:18.636101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.636021    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"]
Oct 31 16:28:18.640095 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:28:18.640066    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d54457_52b4_4cf0_94ab_2e99c2c2a5b1.slice/crio-e64d20c8c6fe56f4a428e158ac4e963df1fa0f16f62d07c347f122eead45d70b WatchSource:0}: Error finding container e64d20c8c6fe56f4a428e158ac4e963df1fa0f16f62d07c347f122eead45d70b: Status 404 returned error can't find the container with id e64d20c8c6fe56f4a428e158ac4e963df1fa0f16f62d07c347f122eead45d70b
Oct 31 16:28:18.905373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:18.905291    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644" event={"ID":"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1","Type":"ContainerStarted","Data":"e64d20c8c6fe56f4a428e158ac4e963df1fa0f16f62d07c347f122eead45d70b"}
Oct 31 16:28:19.915731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:19.915701    4519 generic.go:356] "Generic (PLEG): container finished" podID="a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" containerID="1fe00d9161736b119c815ee7ab9983c7827fbfa977d9401c12087633afbe0fe4" exitCode=0
Oct 31 16:28:19.916181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:19.915746    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644" event={"ID":"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1","Type":"ContainerDied","Data":"1fe00d9161736b119c815ee7ab9983c7827fbfa977d9401c12087633afbe0fe4"}
Oct 31 16:28:21.139859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.139832    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:21.231365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.231335    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca\") pod \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") "
Oct 31 16:28:21.231561 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.231391    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca\") pod \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") "
Oct 31 16:28:21.231561 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.231492    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm\" (UniqueName: \"kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm\") pod \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\" (UID: \"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1\") "
Oct 31 16:28:21.231818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.231794    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca" pod "a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" (UID: "a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:28:21.232260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.232240    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:21.233476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.233438    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca" pod "a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" (UID: "a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:28:21.233683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.233667    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm" pod "a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" (UID: "a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1"). InnerVolumeSpecName "kube-api-access-5msrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:28:21.333394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.333353    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:21.333394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.333385    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5msrm\" (UniqueName: \"kubernetes.io/projected/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1-kube-api-access-5msrm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:21.933791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.933758    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"
Oct 31 16:28:21.933791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.933782    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644" event={"ID":"a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1","Type":"ContainerDied","Data":"e64d20c8c6fe56f4a428e158ac4e963df1fa0f16f62d07c347f122eead45d70b"}
Oct 31 16:28:21.934025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:21.933832    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64d20c8c6fe56f4a428e158ac4e963df1fa0f16f62d07c347f122eead45d70b"
Oct 31 16:28:24.006093 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:28:24.006046    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:28:26.056937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:26.056896    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"]
Oct 31 16:28:26.098698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:26.098670    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9148f6d7-46hjk"]
Oct 31 16:28:26.441637 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:26.441562    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70cc646-f815-4957-a15a-2e6f8aa34529" path="/var/lib/kubelet/pods/c70cc646-f815-4957-a15a-2e6f8aa34529/volumes"
Oct 31 16:28:27.073481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:27.066430    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"]
Oct 31 16:28:27.125733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:27.125696    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc938915-k67gq"]
Oct 31 16:28:28.262928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.262900    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"]
Oct 31 16:28:28.314174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.314134    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a210d9b-rkghn"]
Oct 31 16:28:28.479491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.462003    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/health\": context deadline exceeded"
Oct 31 16:28:28.488582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.488536    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.503358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.488910    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 16:28:28.597630 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.597534    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.621615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.613963    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.653483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.614532    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:28:28.653684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.616057    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.662481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.616085    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 16:28:28.705513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.678483    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.705513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.686119    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:28:28.707065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.707025    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn" podUID="8ac757df-4216-40c2-8219-c01aade2b9f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.735867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.735820    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.769482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.769224    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e" path="/var/lib/kubelet/pods/5547d5cc-7dbc-464e-9fcd-a3e23bb3c46e/volumes"
Oct 31 16:28:28.804495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.785211    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8" path="/var/lib/kubelet/pods/5e6fc860-d3a2-489b-b1d6-0bf22aa3ffa8/volumes"
Oct 31 16:28:28.804495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.786280    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.964751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:28.964712    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:29.493953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.493898    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:29.503481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.495157    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:28:29.597479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.537444    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:29.626713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.626672    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 16:28:29.627589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.627563    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:28:29.662484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.657527    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:29.663060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.663026    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 16:28:29.674896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.665157    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 16:28:29.686447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.686404    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:29.687002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.665472    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:28:29.689301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.687221    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:28:29.689556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.689537    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:28:29.689752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.689736    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:28:29.771655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.767383    4519 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Oct 31 16:28:29.932163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.932101    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:28:29.952475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.952372    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:28:29.956867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:29.956719    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:28:30.007302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:30.007260    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:28:30.009839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:30.009747    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:28:30.021132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:30.020287    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:28:30.049578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:30.049549    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"]
Oct 31 16:28:30.054739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:30.054711    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-677da01e-xshpn"]
Oct 31 16:28:30.441556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:30.441522    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25223c0f-3844-4822-b0cf-11ba4beff2cd" path="/var/lib/kubelet/pods/25223c0f-3844-4822-b0cf-11ba4beff2cd/volumes"
Oct 31 16:28:33.053182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:33.053139    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"]
Oct 31 16:28:33.095351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:33.095314    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79ec77a-wq76n"]
Oct 31 16:28:34.442373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:34.442342    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce328c5d-1aba-4298-99ef-8a06752141e6" path="/var/lib/kubelet/pods/ce328c5d-1aba-4298-99ef-8a06752141e6/volumes"
Oct 31 16:28:37.357326 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.357288    4519 scope.go:122] "RemoveContainer" containerID="c2405d660b0d841dc208f1bd5d77f66dcb976ff17016ae80f7379a725b73bc4d"
Oct 31 16:28:37.370309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.370290    4519 scope.go:122] "RemoveContainer" containerID="2b09c676b69675268bce11674130df08a652a484a283affd53418cf9014d69e4"
Oct 31 16:28:37.380528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.380502    4519 scope.go:122] "RemoveContainer" containerID="ed32cc9c7115a9d34961a8ae9b1198bab91335ba2b392308f3309fdebc35ec2a"
Oct 31 16:28:37.391448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.391425    4519 scope.go:122] "RemoveContainer" containerID="62f244fd93f882f75568090f7dfa71581bac22da4419474f8d220dc3226fb1ff"
Oct 31 16:28:37.402549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.402528    4519 scope.go:122] "RemoveContainer" containerID="e2b7bbb8d62050dfebaf96901faac7e489e54d36cde07e410161d16b6f5378ad"
Oct 31 16:28:37.414572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.414551    4519 scope.go:122] "RemoveContainer" containerID="dc30cb1d73bda110fb9fa9f3f117b2069098c88199891d6f73cc7cdfc0e61605"
Oct 31 16:28:37.424532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:37.424508    4519 scope.go:122] "RemoveContainer" containerID="85313d0d306020460b0263740ec98b42e400ea2029e6a028c322895532b16f1d"
Oct 31 16:28:38.606737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:38.606659    4519 generic.go:356] "Generic (PLEG): container finished" podID="23f8058a-6c74-450b-8481-0e5900e34651" containerID="b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def" exitCode=0
Oct 31 16:28:38.607241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:38.606780    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" event={"ID":"23f8058a-6c74-450b-8481-0e5900e34651","Type":"ContainerDied","Data":"b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"}
Oct 31 16:28:38.607325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:38.607307    4519 scope.go:122] "RemoveContainer" containerID="b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"
Oct 31 16:28:38.609058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:38.609027    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:38.609166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:38.609115    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:38.609166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:38.609154    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:39.050155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:39.050120    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"]
Oct 31 16:28:39.054317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:39.054291    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7456051-hpcxd"]
Oct 31 16:28:39.617090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:39.617055    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" event={"ID":"23f8058a-6c74-450b-8481-0e5900e34651","Type":"ContainerStarted","Data":"1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e"}
Oct 31 16:28:39.823897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:39.823868    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:28:39.824810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:39.824783    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:28:39.824952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:39.824836    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 16:28:40.444902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:40.444865    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9e8cf8-5135-47b3-920c-1eca08513d13" path="/var/lib/kubelet/pods/ca9e8cf8-5135-47b3-920c-1eca08513d13/volumes"
Oct 31 16:28:40.824614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:40.824583    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:28:40.825008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:40.824639    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 16:28:41.050200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.050165    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"]
Oct 31 16:28:41.090534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.089570    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:28:41.097502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.097437    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"]
Oct 31 16:28:41.101721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.101694    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a13574d0-pgbqb"]
Oct 31 16:28:41.105271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.105245    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268ebadf-gb2hb"]
Oct 31 16:28:41.825091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.825053    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:28:41.825638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:41.825134    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 16:28:42.442140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:42.442100    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c5339e-f5f9-443a-8c95-708d39f25b43" path="/var/lib/kubelet/pods/55c5339e-f5f9-443a-8c95-708d39f25b43/volumes"
Oct 31 16:28:42.442724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:42.442703    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4343b8-1166-459c-b363-5d368bdfda3b" path="/var/lib/kubelet/pods/6c4343b8-1166-459c-b363-5d368bdfda3b/volumes"
Oct 31 16:28:42.824252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:42.824215    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:28:42.824425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:42.824287    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 16:28:43.044776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:43.044741    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"]
Oct 31 16:28:43.113560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:43.110223    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c0c9ac9-ldfnd"]
Oct 31 16:28:43.828012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:43.825054    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:28:43.828012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:43.825135    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 16:28:44.440843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:44.440812    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5559663-048a-4e9b-9650-77def958c683" path="/var/lib/kubelet/pods/d5559663-048a-4e9b-9650-77def958c683/volumes"
Oct 31 16:28:44.828349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:44.828319    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:28:44.829657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:44.829637    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:28:53.413120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:28:53.413037    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:28:54.005649 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:28:54.005614    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:29:04.048872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.048837    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"]
Oct 31 16:29:04.095764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.095726    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6569ee9e-c6kll"]
Oct 31 16:29:04.441869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.441775    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c819957-a750-4960-a8bb-fcb5f0da745f" path="/var/lib/kubelet/pods/1c819957-a750-4960-a8bb-fcb5f0da745f/volumes"
Oct 31 16:29:04.898299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.898267    4519 generic.go:356] "Generic (PLEG): container finished" podID="821daccc-0a90-4511-9c5e-613589b2681c" containerID="2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c" exitCode=0
Oct 31 16:29:04.898515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.898343    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" event={"ID":"821daccc-0a90-4511-9c5e-613589b2681c","Type":"ContainerDied","Data":"2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"}
Oct 31 16:29:04.898984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.898956    4519 scope.go:122] "RemoveContainer" containerID="2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"
Oct 31 16:29:04.901169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.901141    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:04.901273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.901208    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:04.901273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:04.901241    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:05.908664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:05.908625    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" event={"ID":"821daccc-0a90-4511-9c5e-613589b2681c","Type":"ContainerStarted","Data":"5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06"}
Oct 31 16:29:06.259243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:06.259210    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:29:06.325884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:06.325853    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:29:06.327702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:06.327671    4519 patch_prober.go:29] interesting pod/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 16:29:06.327954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:06.327921    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 16:29:06.494691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:06.494652    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"]
Oct 31 16:29:06.498784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:06.498752    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1bb9544-pz7ft"]
Oct 31 16:29:07.325104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:07.325065    4519 patch_prober.go:29] interesting pod/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 16:29:07.325567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:07.325120    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 16:29:08.338895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:08.338850    4519 patch_prober.go:29] interesting pod/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 16:29:08.339540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:08.339501    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 16:29:08.442164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:08.442129    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b1ff9c-1db2-451e-88f0-adc1f3daf14f" path="/var/lib/kubelet/pods/c8b1ff9c-1db2-451e-88f0-adc1f3daf14f/volumes"
Oct 31 16:29:09.334690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:09.334640    4519 patch_prober.go:29] interesting pod/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 16:29:09.334920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:09.334736    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 16:29:10.326022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:10.325984    4519 patch_prober.go:29] interesting pod/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 16:29:10.326570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:10.326050    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 16:29:11.325819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:11.325788    4519 patch_prober.go:29] interesting pod/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 31 16:29:11.326142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:11.326112    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 31 16:29:12.330303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:12.330271    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:29:12.331497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:12.331476    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:29:13.335601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:13.335516    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"]
Oct 31 16:29:13.382716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:13.382680    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ce4664d0-tr644"]
Oct 31 16:29:14.441313 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:14.441280    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" path="/var/lib/kubelet/pods/a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1/volumes"
Oct 31 16:29:16.364712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.364674    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"]
Oct 31 16:29:16.365702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.365669    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" containerName="revoke-keys"
Oct 31 16:29:16.365702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.365691    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" containerName="revoke-keys"
Oct 31 16:29:16.365902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.365886    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d54457-52b4-4cf0-94ab-2e99c2c2a5b1" containerName="revoke-keys"
Oct 31 16:29:16.371210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.371186    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.452439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.452408    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"]
Oct 31 16:29:16.470315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.470286    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dac4fa-openshift-default-dockercfg-rb5sv\""
Oct 31 16:29:16.492228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.492191    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgnzx\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.492941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.492916    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.493557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.493522    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.493822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.493802    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.494011 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.493997    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.494214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.494198    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.494403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.494386    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.494635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.494618    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.494815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.494802    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.639675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.639588    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.640843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.640819    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.641656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.640705    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.641296    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.642237    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.672477    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.672528    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.672606    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.672730    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.672812    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.672947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.672904    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgnzx\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.686243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.673915    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.686243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.674594    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.686243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.674828    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.705030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.702891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.710257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.708402    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.736950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.736681    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.740884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.739979    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgnzx\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx\") pod \"e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") " pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:16.990834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:16.990780    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:17.451704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.451621    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"]
Oct 31 16:29:17.495871 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:29:17.495839    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38d95c8_7437_4a33_8270_1684c798cd86.slice/crio-21fdc6b84c325350342b833dfcb35311a7a1b24b5b316613056132456f116df8 WatchSource:0}: Error finding container 21fdc6b84c325350342b833dfcb35311a7a1b24b5b316613056132456f116df8: Status 404 returned error can't find the container with id 21fdc6b84c325350342b833dfcb35311a7a1b24b5b316613056132456f116df8
Oct 31 16:29:17.498660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.498623    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:17.498772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.498708    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:17.498772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.498758    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:17.694512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.694437    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"]
Oct 31 16:29:17.699841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.699816    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.753781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.750544    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-f43174-openshift-default-dockercfg-9wjgc\""
Oct 31 16:29:17.761506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.756391    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"]
Oct 31 16:29:17.794862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.794827    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.794882    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795020    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795178    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdkp\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795358    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795485    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795614    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.795709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.795686    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897193 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897158    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897226    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897292    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdkp\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897335    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897378    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897506    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897602    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897622    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897667    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897730    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897765    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.897811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.897785    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.898033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.898007    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.898149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.898129    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.900012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.899992    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.901779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.901759    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.952889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.952847    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:17.978478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:17.976944    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdkp\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:18.017474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.015729    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:18.204150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.201829    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" event={"ID":"f38d95c8-7437-4a33-8270-1684c798cd86","Type":"ContainerStarted","Data":"7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"}
Oct 31 16:29:18.204150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.201901    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" event={"ID":"f38d95c8-7437-4a33-8270-1684c798cd86","Type":"ContainerStarted","Data":"21fdc6b84c325350342b833dfcb35311a7a1b24b5b316613056132456f116df8"}
Oct 31 16:29:18.316938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.316844    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podStartSLOduration=2.316829605 podStartE2EDuration="2.316829605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:18.313514389 +0000 UTC m=+1602.013443709" watchObservedRunningTime="2026-10-31 16:29:18.316829605 +0000 UTC m=+1602.016758915"
Oct 31 16:29:18.448764 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:29:18.448727    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a758d2_3317_4ee9_bb46_39e7c5476698.slice/crio-35904d346743c914eeba03f762dbb94ecf3cab51e0a880cbb0ff03d7d67cd429 WatchSource:0}: Error finding container 35904d346743c914eeba03f762dbb94ecf3cab51e0a880cbb0ff03d7d67cd429: Status 404 returned error can't find the container with id 35904d346743c914eeba03f762dbb94ecf3cab51e0a880cbb0ff03d7d67cd429
Oct 31 16:29:18.454930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.454870    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:18.455361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.454967    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:18.455361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.455012    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:18.461742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.460582    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"]
Oct 31 16:29:18.991116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.991088    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:18.997041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:18.997016    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:19.211530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:19.211494    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" event={"ID":"b5a758d2-3317-4ee9-bb46-39e7c5476698","Type":"ContainerStarted","Data":"d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"}
Oct 31 16:29:19.211731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:19.211553    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" event={"ID":"b5a758d2-3317-4ee9-bb46-39e7c5476698","Type":"ContainerStarted","Data":"35904d346743c914eeba03f762dbb94ecf3cab51e0a880cbb0ff03d7d67cd429"}
Oct 31 16:29:19.211899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:19.211883    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:19.212791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:19.212771    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:29:19.325551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:19.325416    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podStartSLOduration=2.325396463 podStartE2EDuration="2.325396463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:19.321758841 +0000 UTC m=+1603.021688154" watchObservedRunningTime="2026-10-31 16:29:19.325396463 +0000 UTC m=+1603.025325774"
Oct 31 16:29:20.022053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:20.021906    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:20.045908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:20.045875    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:20.222274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:20.222242    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:20.222897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:20.222865    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:29:21.891603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:21.891564    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"]
Oct 31 16:29:21.892219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:21.892104    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy" containerID="cri-o://8eeaeb604a6b8eb647b71ba87d5f30e07608f5f8654a8f65bfe88eb008b7de01" gracePeriod=30
Oct 31 16:29:21.994878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:21.994845    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"]
Oct 31 16:29:21.995100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:21.995058    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy" containerID="cri-o://d1f7bea4293db9d76ebc2ea4395e487be39a9c08c87f825ca0012db955b2b713" gracePeriod=30
Oct 31 16:29:24.006049 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:29:24.006010    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:29:27.294563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.294524    4519 generic.go:356] "Generic (PLEG): container finished" podID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerID="8eeaeb604a6b8eb647b71ba87d5f30e07608f5f8654a8f65bfe88eb008b7de01" exitCode=0
Oct 31 16:29:27.295379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.295145    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" event={"ID":"087a6981-f457-48a6-b9a2-5e57e7ec351a","Type":"ContainerDied","Data":"8eeaeb604a6b8eb647b71ba87d5f30e07608f5f8654a8f65bfe88eb008b7de01"}
Oct 31 16:29:27.309332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.309294    4519 generic.go:356] "Generic (PLEG): container finished" podID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerID="d1f7bea4293db9d76ebc2ea4395e487be39a9c08c87f825ca0012db955b2b713" exitCode=0
Oct 31 16:29:27.309789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.309572    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" event={"ID":"014bdfbf-4d83-45f1-8b5f-302c38a61565","Type":"ContainerDied","Data":"d1f7bea4293db9d76ebc2ea4395e487be39a9c08c87f825ca0012db955b2b713"}
Oct 31 16:29:27.424603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.424574    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:29:27.451635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.451607    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:29:27.523975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.523878    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524021    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524059    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524103    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524202    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524265    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524294    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524348    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524392    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524424    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524563    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524627    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524656    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524764    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t\") pod \"014bdfbf-4d83-45f1-8b5f-302c38a61565\" (UID: \"014bdfbf-4d83-45f1-8b5f-302c38a61565\") "
Oct 31 16:29:27.524966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524809    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.524966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.524887    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.525080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.525005    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token\") pod \"087a6981-f457-48a6-b9a2-5e57e7ec351a\" (UID: \"087a6981-f457-48a6-b9a2-5e57e7ec351a\") "
Oct 31 16:29:27.525474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.525414    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.525677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.525654    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:27.525865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.525841    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.526413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.526079    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.526413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.526122    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.526639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.526313    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.526707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.526632    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.527098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.527033    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:27.527297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.527198    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.527297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.526237    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.527297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.527238    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087a6981-f457-48a6-b9a2-5e57e7ec351a-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.527297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.527235    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.538890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.538835    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "kube-api-access-mjs7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:27.545901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.545863    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.555929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.555878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:27.558035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.555942    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "kube-api-access-xm44t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:27.558420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.558360    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo" pod "087a6981-f457-48a6-b9a2-5e57e7ec351a" (UID: "087a6981-f457-48a6-b9a2-5e57e7ec351a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:27.560748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.560687    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:27.561435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.561402    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:27.561929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.561893    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy" pod "014bdfbf-4d83-45f1-8b5f-302c38a61565" (UID: "014bdfbf-4d83-45f1-8b5f-302c38a61565"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.629966    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630007    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630023    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014bdfbf-4d83-45f1-8b5f-302c38a61565-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630038    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630054    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630078    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630092    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630107    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630120    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630133    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630147    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630161    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm44t\" (UniqueName: \"kubernetes.io/projected/014bdfbf-4d83-45f1-8b5f-302c38a61565-kube-api-access-xm44t\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630177    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630190    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjs7w\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-kube-api-access-mjs7w\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630204    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087a6981-f457-48a6-b9a2-5e57e7ec351a-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:27.636516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:27.630218    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014bdfbf-4d83-45f1-8b5f-302c38a61565-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.318956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.318876    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"
Oct 31 16:29:28.318956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.318897    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp" event={"ID":"087a6981-f457-48a6-b9a2-5e57e7ec351a","Type":"ContainerDied","Data":"1b3bc0596167fc39376bed9d54c8f361e50301ad2a93ad90d9a5a969ffabe2f8"}
Oct 31 16:29:28.319937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.318989    4519 scope.go:122] "RemoveContainer" containerID="8eeaeb604a6b8eb647b71ba87d5f30e07608f5f8654a8f65bfe88eb008b7de01"
Oct 31 16:29:28.325796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.325764    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx" event={"ID":"014bdfbf-4d83-45f1-8b5f-302c38a61565","Type":"ContainerDied","Data":"ae939ebbc5251a8cd86bb969ea3af2c4071f94c19cf834e390f159dbf86a75c5"}
Oct 31 16:29:28.325998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.325909    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"
Oct 31 16:29:28.341303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.341278    4519 scope.go:122] "RemoveContainer" containerID="d1f7bea4293db9d76ebc2ea4395e487be39a9c08c87f825ca0012db955b2b713"
Oct 31 16:29:28.530969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.530932    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"]
Oct 31 16:29:28.574424 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.574354    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-2mtxp"]
Oct 31 16:29:28.680886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.680856    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"]
Oct 31 16:29:28.685086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:28.685059    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-89bcx"]
Oct 31 16:29:30.440371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:30.440336    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" path="/var/lib/kubelet/pods/014bdfbf-4d83-45f1-8b5f-302c38a61565/volumes"
Oct 31 16:29:30.440952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:30.440934    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" path="/var/lib/kubelet/pods/087a6981-f457-48a6-b9a2-5e57e7ec351a/volumes"
Oct 31 16:29:33.742725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.742690    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"]
Oct 31 16:29:33.743508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.743485    4519 cpu_manager.go:414] "containerMap: removing container" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy"
Oct 31 16:29:33.743508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.743502    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy"
Oct 31 16:29:33.743595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.743521    4519 cpu_manager.go:414] "containerMap: removing container" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy"
Oct 31 16:29:33.743595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.743527    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy"
Oct 31 16:29:33.743659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.743642    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="014bdfbf-4d83-45f1-8b5f-302c38a61565" containerName="istio-proxy"
Oct 31 16:29:33.743659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.743654    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="087a6981-f457-48a6-b9a2-5e57e7ec351a" containerName="istio-proxy"
Oct 31 16:29:33.749023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.749005    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.798838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.798808    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"]
Oct 31 16:29:33.839138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.839106    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.839329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.839261    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.839329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.839299    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglnl\" (UniqueName: \"kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.840870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.840831    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:29:33.840991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.840877    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:29:33.840991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.840895    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:29:33.840991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.840950    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:29:33.940996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.940963    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.941194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.941071    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.941194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.941100    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglnl\" (UniqueName: \"kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.941876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.941849    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.943558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.943540    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:33.995246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:33.995170    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglnl\" (UniqueName: \"kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl\") pod \"maas-api-revoke-keys-e2e-mt-a7907d70-vprwp\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:34.067939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:34.067904    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:34.441128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:34.441040    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"]
Oct 31 16:29:34.445920 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:29:34.445895    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fa7860_cd5b_489a_97cf_0c7da28c22b0.slice/crio-d6910524c86b587bd81fc5399b46dd95a19565c5b46063197a7042d0b9dbd89b WatchSource:0}: Error finding container d6910524c86b587bd81fc5399b46dd95a19565c5b46063197a7042d0b9dbd89b: Status 404 returned error can't find the container with id d6910524c86b587bd81fc5399b46dd95a19565c5b46063197a7042d0b9dbd89b
Oct 31 16:29:35.451478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:35.441201    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp" event={"ID":"40fa7860-cd5b-489a-97cf-0c7da28c22b0","Type":"ContainerStarted","Data":"4453d3b318423c36e492b45beecc4e67d99d4b8d81b85b836b192d0caf3ce2e2"}
Oct 31 16:29:35.451478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:35.441277    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp" event={"ID":"40fa7860-cd5b-489a-97cf-0c7da28c22b0","Type":"ContainerStarted","Data":"d6910524c86b587bd81fc5399b46dd95a19565c5b46063197a7042d0b9dbd89b"}
Oct 31 16:29:35.616819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:35.616771    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp" podStartSLOduration=2.616756177 podStartE2EDuration="2.616756177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:35.614735505 +0000 UTC m=+1619.314664817" watchObservedRunningTime="2026-10-31 16:29:35.616756177 +0000 UTC m=+1619.316685503"
Oct 31 16:29:36.458351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:36.458307    4519 generic.go:356] "Generic (PLEG): container finished" podID="40fa7860-cd5b-489a-97cf-0c7da28c22b0" containerID="4453d3b318423c36e492b45beecc4e67d99d4b8d81b85b836b192d0caf3ce2e2" exitCode=0
Oct 31 16:29:36.458738 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:36.458433    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp" event={"ID":"40fa7860-cd5b-489a-97cf-0c7da28c22b0","Type":"ContainerDied","Data":"4453d3b318423c36e492b45beecc4e67d99d4b8d81b85b836b192d0caf3ce2e2"}
Oct 31 16:29:37.500982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.500949    4519 scope.go:122] "RemoveContainer" containerID="b8ca320daf40f9b51ce9f938a05b1c0491c11af42166790af1a53130b9bd4b9b"
Oct 31 16:29:37.512443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.512421    4519 scope.go:122] "RemoveContainer" containerID="5f8136305e890d99394da236f4657826532e2af68eb2fefdde0426eca8a132b2"
Oct 31 16:29:37.522225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.522204    4519 scope.go:122] "RemoveContainer" containerID="94a4fc2ea983d5c7ef7e7ee6861f0b91c991859254657d1615414dc753617f99"
Oct 31 16:29:37.534553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.534529    4519 scope.go:122] "RemoveContainer" containerID="adc387ad69f85c4ce64edc08a6594c449d12c6f22b573ad835b5d66327aa9256"
Oct 31 16:29:37.545052 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.545031    4519 scope.go:122] "RemoveContainer" containerID="94d6e92f1115a77682d5d944a888605fc0dbc1804f57ba241e14f5f0c1650970"
Oct 31 16:29:37.555391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.555305    4519 scope.go:122] "RemoveContainer" containerID="badbeb105513df1aa3e8f5c7a53faf2a3fb31932088e6680455e949af29355f2"
Oct 31 16:29:37.599036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.597576    4519 scope.go:122] "RemoveContainer" containerID="98e7f2a80e61747f65b16ca7da26e4cd371660631b920a8168de7c5d1a2e79b5"
Oct 31 16:29:37.622051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.622025    4519 scope.go:122] "RemoveContainer" containerID="2f73421dfbbf72b0954d888800c058e3dfcf83ea1abf154241790e7db3395cf2"
Oct 31 16:29:37.637824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.637682    4519 scope.go:122] "RemoveContainer" containerID="64f40d3c9161a6cfe0990f75c951697520c720c188d7c8782bc4fd14c369a63d"
Oct 31 16:29:37.652117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.652094    4519 scope.go:122] "RemoveContainer" containerID="b2b02216dc5929bc13a5f369e5265bcda479f802b3317753808197dac7d88fb9"
Oct 31 16:29:37.692324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.691873    4519 scope.go:122] "RemoveContainer" containerID="7c08785f67ce3c509b120ce72c8818425b74a9de701c01180601dc7c96472c25"
Oct 31 16:29:37.731601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.731569    4519 scope.go:122] "RemoveContainer" containerID="a28e9d8344099c25df70b729bc23899ac40a8e1193a7269d6b371e549bd62045"
Oct 31 16:29:37.732792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.732772    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:37.741762 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.741740    4519 scope.go:122] "RemoveContainer" containerID="6ce20f192379ad43dd6e79cbd21bab48fcb80c32142c78cc17a2b0e0897e789b"
Oct 31 16:29:37.752111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.752090    4519 scope.go:122] "RemoveContainer" containerID="7a77e10cec34c42c9257568a38b0a70ba0bb1291939e7195dbca8098732457db"
Oct 31 16:29:37.792445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.792425    4519 scope.go:122] "RemoveContainer" containerID="bfbc441758636bb2a15e64e728ece0a4807a6f23a964f5229bea6438c89692e9"
Oct 31 16:29:37.804051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.804030    4519 scope.go:122] "RemoveContainer" containerID="c80ed7904352ff6cb25410d5799b8dfd162a28f97019e40b49198873068de289"
Oct 31 16:29:37.814081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.814049    4519 scope.go:122] "RemoveContainer" containerID="35df525b64813c08a73701dced17cf1cc49a953afafc96d953a17adc3668278c"
Oct 31 16:29:37.850242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.850215    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl\" (UniqueName: \"kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl\") pod \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") "
Oct 31 16:29:37.850612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.850590    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca\") pod \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") "
Oct 31 16:29:37.850772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.850631    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca\") pod \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\" (UID: \"40fa7860-cd5b-489a-97cf-0c7da28c22b0\") "
Oct 31 16:29:37.852888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.852860    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca" pod "40fa7860-cd5b-489a-97cf-0c7da28c22b0" (UID: "40fa7860-cd5b-489a-97cf-0c7da28c22b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:37.855508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.855435    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca" pod "40fa7860-cd5b-489a-97cf-0c7da28c22b0" (UID: "40fa7860-cd5b-489a-97cf-0c7da28c22b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:37.855596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.855556    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl" pod "40fa7860-cd5b-489a-97cf-0c7da28c22b0" (UID: "40fa7860-cd5b-489a-97cf-0c7da28c22b0"). InnerVolumeSpecName "kube-api-access-nglnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:37.952715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.952682    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:37.952715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.952721    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fa7860-cd5b-489a-97cf-0c7da28c22b0-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:37.952885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:37.952736    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglnl\" (UniqueName: \"kubernetes.io/projected/40fa7860-cd5b-489a-97cf-0c7da28c22b0-kube-api-access-nglnl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:38.502661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:38.502632    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"
Oct 31 16:29:38.503009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:38.502656    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp" event={"ID":"40fa7860-cd5b-489a-97cf-0c7da28c22b0","Type":"ContainerDied","Data":"d6910524c86b587bd81fc5399b46dd95a19565c5b46063197a7042d0b9dbd89b"}
Oct 31 16:29:38.503009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:38.502711    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6910524c86b587bd81fc5399b46dd95a19565c5b46063197a7042d0b9dbd89b"
Oct 31 16:29:46.023841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.023805    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"]
Oct 31 16:29:46.581782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.581681    4519 generic.go:356] "Generic (PLEG): container finished" podID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerID="76b24565c90e9a45bbf3017326f917216b3f7a4a5f6b507013e55d3053b298cc" exitCode=0
Oct 31 16:29:46.581782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.581764    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" event={"ID":"40d9cc78-f6fb-4031-a540-adc0d2e4fadd","Type":"ContainerDied","Data":"76b24565c90e9a45bbf3017326f917216b3f7a4a5f6b507013e55d3053b298cc"}
Oct 31 16:29:46.582311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.582292    4519 scope.go:122] "RemoveContainer" containerID="76b24565c90e9a45bbf3017326f917216b3f7a4a5f6b507013e55d3053b298cc"
Oct 31 16:29:46.584008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.583977    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:46.584097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.584048    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:46.584097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:46.584086    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:47.613790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:47.613749    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" event={"ID":"40d9cc78-f6fb-4031-a540-adc0d2e4fadd","Type":"ContainerStarted","Data":"985ce94d616582c21d6c88379c8e3f82a06c0c2106ad6cdabdd48944e6f66f78"}
Oct 31 16:29:47.655527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:47.636948    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:29:47.657696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:47.656801    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 16:29:47.657928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:47.657904    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 16:29:48.651397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:48.650133    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 16:29:48.651397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:48.650236    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 16:29:49.635996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:49.635963    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 16:29:49.636179 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:49.636025    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 16:29:50.636937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:50.636894    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 16:29:50.637427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:50.636953    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 16:29:51.024597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.024557    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n" podUID="d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" containerName="payload-pre-processing" containerID="cri-o://ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29" gracePeriod=25
Oct 31 16:29:51.356584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.356553    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:29:51.517363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.517327    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume\") pod \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") "
Oct 31 16:29:51.517613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.517386    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k\" (UniqueName: \"kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k\") pod \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\" (UID: \"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564\") "
Oct 31 16:29:51.517882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.517860    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume" pod "d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" (UID: "d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:51.518237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.518219    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:51.519629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.519607    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k" pod "d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" (UID: "d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564"). InnerVolumeSpecName "kube-api-access-d2x8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:51.619809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.619728    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2x8k\" (UniqueName: \"kubernetes.io/projected/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564-kube-api-access-d2x8k\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:51.636151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.636123    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 16:29:51.636297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.636179    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 16:29:51.709391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.709360    4519 generic.go:356] "Generic (PLEG): container finished" podID="d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" containerID="ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29" exitCode=0
Oct 31 16:29:51.709900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.709403    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"
Oct 31 16:29:51.709900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.709435    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n" event={"ID":"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564","Type":"ContainerDied","Data":"ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29"}
Oct 31 16:29:51.709900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.709504    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n" event={"ID":"d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564","Type":"ContainerDied","Data":"f73721841a72720cd58c92d81a80c2e7ffe6afec543247818661dc8dfc09f9be"}
Oct 31 16:29:51.709900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.709531    4519 scope.go:122] "RemoveContainer" containerID="ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29"
Oct 31 16:29:51.722204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.722184    4519 scope.go:122] "RemoveContainer" containerID="ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29"
Oct 31 16:29:51.722477 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:29:51.722444    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29\": container with ID starting with ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29 not found: ID does not exist" containerID="ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29"
Oct 31 16:29:51.722559 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.722487    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29"} err="failed to get container status \"ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29\": rpc error: code = NotFound desc = could not find container \"ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29\": container with ID starting with ff2475d52d5228691b1fdae77a8a4fb227795ac23cb516997dcec92cfdc3ac29 not found: ID does not exist"
Oct 31 16:29:51.894739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.894664    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"]
Oct 31 16:29:51.899009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:51.898983    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a7907d70-85d6dd5999-2c79n"]
Oct 31 16:29:52.682476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:52.680099    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 16:29:52.682476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:52.680163    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 16:29:52.842642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:52.842613    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" path="/var/lib/kubelet/pods/d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564/volumes"
Oct 31 16:29:52.843474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:52.843443    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:29:53.799568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:53.799539    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:29:53.856404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:29:53.856377    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:29:54.005284 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:29:54.005249    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:30:00.193364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.193323    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"]
Oct 31 16:30:00.195304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.195273    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40fa7860-cd5b-489a-97cf-0c7da28c22b0" containerName="revoke-keys"
Oct 31 16:30:00.195304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.195304    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fa7860-cd5b-489a-97cf-0c7da28c22b0" containerName="revoke-keys"
Oct 31 16:30:00.195590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.195352    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" containerName="payload-pre-processing"
Oct 31 16:30:00.195590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.195360    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" containerName="payload-pre-processing"
Oct 31 16:30:00.195702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.195644    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fa7860-cd5b-489a-97cf-0c7da28c22b0" containerName="revoke-keys"
Oct 31 16:30:00.195702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.195671    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b5bcd9-dfe5-4f81-89d6-d33bcdda3564" containerName="payload-pre-processing"
Oct 31 16:30:00.200003 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.199976    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"]
Oct 31 16:30:00.200274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.200117    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.206005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.205980    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"]
Oct 31 16:30:00.206269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.206108    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.212804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.212639    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.222168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.222139    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"]
Oct 31 16:30:00.230756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.230714    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"]
Oct 31 16:30:00.232540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.232515    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.236499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.236476    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"]
Oct 31 16:30:00.236853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.236837    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.242141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.241916    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"]
Oct 31 16:30:00.242141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.242056    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.248504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.248441    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.255250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.255228    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:30:00.255468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.255238    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:30:00.255619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.255261    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:30:00.257193 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.257169    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"]
Oct 31 16:30:00.261771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.261748    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"]
Oct 31 16:30:00.266116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.266094    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"]
Oct 31 16:30:00.274741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.274717    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmvs\" (UniqueName: \"kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.274857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.274771    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.274925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.274866    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.275010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.274984    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4lv\" (UniqueName: \"kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.304236 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.304203    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"]
Oct 31 16:30:00.312536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.312504    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"]
Oct 31 16:30:00.359255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.359218    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"]
Oct 31 16:30:00.364508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.363986    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"]
Oct 31 16:30:00.376041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376012    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.376244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376069    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmvs\" (UniqueName: \"kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.376244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376193    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqnn\" (UniqueName: \"kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.376375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376275    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.376375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376355    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.376527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376395    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.376527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376429    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtmf\" (UniqueName: \"kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf\") pod \"maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.376527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376491    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4cm\" (UniqueName: \"kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm\") pod \"maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.376652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376550    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.376652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376619    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65t7n\" (UniqueName: \"kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.376719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376650    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4lv\" (UniqueName: \"kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.376916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376896    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gtv\" (UniqueName: \"kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.376963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.376946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.377031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.377014    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.377241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.377221    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.377287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.377227    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.430832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.430785    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmvs\" (UniqueName: \"kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs\") pod \"maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.431723 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.431693    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4lv\" (UniqueName: \"kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.478021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.477990    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.478218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478044    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtmf\" (UniqueName: \"kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf\") pod \"maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.478218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478077    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4cm\" (UniqueName: \"kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm\") pod \"maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.478218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478101    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.478218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478128    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65t7n\" (UniqueName: \"kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.478425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478370    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gtv\" (UniqueName: \"kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.478425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478414    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.478562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478531    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.478638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478620    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.478729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478712    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqnn\" (UniqueName: \"kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.478903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.478881    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.479112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.479083    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.479232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.479211    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.479312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.479283    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.479363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.479346    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.524582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.524549    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:00.531937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.531912    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gtv\" (UniqueName: \"kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.532215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.532190    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqnn\" (UniqueName: \"kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.532340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.532300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtmf\" (UniqueName: \"kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf\") pod \"maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:00.532509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.532483    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65t7n\" (UniqueName: \"kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n\") pod \"maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.533255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.533235    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4cm\" (UniqueName: \"kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm\") pod \"maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.548027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.539812    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:00.553774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.553442    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:00.582686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.571831    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:00.582686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.572838    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:00.613493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.588122    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:00.613493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:00.592222    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:01.578078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.577981    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"]
Oct 31 16:30:01.583349 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.583317    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc30e44_dfd2_417c_83e2_b0b3c5331259.slice/crio-3258c9db4ef00bac73b09a4475ffb23189a3a6099dd931daf54536f3f9d44b4e WatchSource:0}: Error finding container 3258c9db4ef00bac73b09a4475ffb23189a3a6099dd931daf54536f3f9d44b4e: Status 404 returned error can't find the container with id 3258c9db4ef00bac73b09a4475ffb23189a3a6099dd931daf54536f3f9d44b4e
Oct 31 16:30:01.682297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.682239    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"]
Oct 31 16:30:01.702577 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.702553    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"]
Oct 31 16:30:01.709305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.709276    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"]
Oct 31 16:30:01.735153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.735125    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"]
Oct 31 16:30:01.740357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.740335    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"]
Oct 31 16:30:01.768792 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.768749    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36aa932f_7edf_4ac1_9a5f_f0f5ba6d7679.slice/crio-8f5bc262e3afffcf11297b3fc978d5e31c8406ce5d63848dc1d6b9ad944d6e1c WatchSource:0}: Error finding container 8f5bc262e3afffcf11297b3fc978d5e31c8406ce5d63848dc1d6b9ad944d6e1c: Status 404 returned error can't find the container with id 8f5bc262e3afffcf11297b3fc978d5e31c8406ce5d63848dc1d6b9ad944d6e1c
Oct 31 16:30:01.770090 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.770055    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5af6878_2c2c_48f8_9753_a6da970432d0.slice/crio-9b62c08d1ed4c11f997ce5509a69fc81b8452eb0e95df090b3e20d3bd2a99349 WatchSource:0}: Error finding container 9b62c08d1ed4c11f997ce5509a69fc81b8452eb0e95df090b3e20d3bd2a99349: Status 404 returned error can't find the container with id 9b62c08d1ed4c11f997ce5509a69fc81b8452eb0e95df090b3e20d3bd2a99349
Oct 31 16:30:01.771125 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.771099    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0801f85_c125_4608_bb37_a49eb4eab201.slice/crio-cdbf0f48ff365d31f6f64d0cfe0d673841b33799b53814b7ed17ec1467fea566 WatchSource:0}: Error finding container cdbf0f48ff365d31f6f64d0cfe0d673841b33799b53814b7ed17ec1467fea566: Status 404 returned error can't find the container with id cdbf0f48ff365d31f6f64d0cfe0d673841b33799b53814b7ed17ec1467fea566
Oct 31 16:30:01.772435 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.772410    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda385b3bc_e028_4510_bdee_5265ce04f4a2.slice/crio-a95ad963d09c53a922a0e4aa4af47d0e3ec22bc2e0abd29e8ec80df6f3f1214b WatchSource:0}: Error finding container a95ad963d09c53a922a0e4aa4af47d0e3ec22bc2e0abd29e8ec80df6f3f1214b: Status 404 returned error can't find the container with id a95ad963d09c53a922a0e4aa4af47d0e3ec22bc2e0abd29e8ec80df6f3f1214b
Oct 31 16:30:01.773635 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.773601    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959573d1_3e1b_4868_acce_a7048f8e7ec3.slice/crio-cbde54b608a484d61831fbf5b88a3c963ac39b46b9a7d6ba28508b1d9725a20a WatchSource:0}: Error finding container cbde54b608a484d61831fbf5b88a3c963ac39b46b9a7d6ba28508b1d9725a20a: Status 404 returned error can't find the container with id cbde54b608a484d61831fbf5b88a3c963ac39b46b9a7d6ba28508b1d9725a20a
Oct 31 16:30:01.815168 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:01.815135    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"]
Oct 31 16:30:01.821044 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:30:01.821014    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337a5581_c4a2_4cfb_91ac_349830a1d098.slice/crio-03bab21f3abaaef030325d8e40af3c44ec82922605c709c4234da81b19a7c3f0 WatchSource:0}: Error finding container 03bab21f3abaaef030325d8e40af3c44ec82922605c709c4234da81b19a7c3f0: Status 404 returned error can't find the container with id 03bab21f3abaaef030325d8e40af3c44ec82922605c709c4234da81b19a7c3f0
Oct 31 16:30:02.266958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.266909    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerStarted","Data":"b572f2b160c0b428ec06f5b3d680fd4de662276c1901d354640c032fea041580"}
Oct 31 16:30:02.267298 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.266984    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerStarted","Data":"03bab21f3abaaef030325d8e40af3c44ec82922605c709c4234da81b19a7c3f0"}
Oct 31 16:30:02.268951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.268920    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" event={"ID":"c0801f85-c125-4608-bb37-a49eb4eab201","Type":"ContainerStarted","Data":"cdbf0f48ff365d31f6f64d0cfe0d673841b33799b53814b7ed17ec1467fea566"}
Oct 31 16:30:02.272253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.272210    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerStarted","Data":"4882d569ce5e6060ac14abd82f0f20d29a4fed62395c3a9ae59620be6ee8703b"}
Oct 31 16:30:02.272394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.272280    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerStarted","Data":"3258c9db4ef00bac73b09a4475ffb23189a3a6099dd931daf54536f3f9d44b4e"}
Oct 31 16:30:02.274110 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.274084    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerStarted","Data":"8f5bc262e3afffcf11297b3fc978d5e31c8406ce5d63848dc1d6b9ad944d6e1c"}
Oct 31 16:30:02.276385 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.276363    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerStarted","Data":"9b62c08d1ed4c11f997ce5509a69fc81b8452eb0e95df090b3e20d3bd2a99349"}
Oct 31 16:30:02.277881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.277817    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerStarted","Data":"cbde54b608a484d61831fbf5b88a3c963ac39b46b9a7d6ba28508b1d9725a20a"}
Oct 31 16:30:02.280288 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.280260    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerStarted","Data":"37d1bd10675dded399b529b2dd214f8872de05a3c3b619dcba71a053836691c7"}
Oct 31 16:30:02.280398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.280318    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerStarted","Data":"a95ad963d09c53a922a0e4aa4af47d0e3ec22bc2e0abd29e8ec80df6f3f1214b"}
Oct 31 16:30:02.376089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.376038    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" podStartSLOduration=2.376018905 podStartE2EDuration="2.376018905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:02.374094599 +0000 UTC m=+1646.074023905" watchObservedRunningTime="2026-10-31 16:30:02.376018905 +0000 UTC m=+1646.075948216"
Oct 31 16:30:02.481700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.481646    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" podStartSLOduration=2.481630907 podStartE2EDuration="2.481630907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:02.479797526 +0000 UTC m=+1646.179726839" watchObservedRunningTime="2026-10-31 16:30:02.481630907 +0000 UTC m=+1646.181560218"
Oct 31 16:30:02.588180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:02.588127    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" podStartSLOduration=2.5881137389999997 podStartE2EDuration="2.588113739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:02.58578653 +0000 UTC m=+1646.285715841" watchObservedRunningTime="2026-10-31 16:30:02.588113739 +0000 UTC m=+1646.288043049"
Oct 31 16:30:03.290950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.290629    4519 generic.go:356] "Generic (PLEG): container finished" podID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerID="dd09f4c8a298d2435c07080b5ce3c962966a02765f090aa65f212bbb94992249" exitCode=60
Oct 31 16:30:03.290950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.290664    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerDied","Data":"dd09f4c8a298d2435c07080b5ce3c962966a02765f090aa65f212bbb94992249"}
Oct 31 16:30:03.291208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.291165    4519 scope.go:122] "RemoveContainer" containerID="dd09f4c8a298d2435c07080b5ce3c962966a02765f090aa65f212bbb94992249"
Oct 31 16:30:03.292571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.292544    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerID="5f19d1be9d24606f1b7b39b8056fad94be75e06b2622cfa3674c9fc798da8954" exitCode=60
Oct 31 16:30:03.292648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.292620    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerDied","Data":"5f19d1be9d24606f1b7b39b8056fad94be75e06b2622cfa3674c9fc798da8954"}
Oct 31 16:30:03.292867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.292853    4519 scope.go:122] "RemoveContainer" containerID="5f19d1be9d24606f1b7b39b8056fad94be75e06b2622cfa3674c9fc798da8954"
Oct 31 16:30:03.294394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.294374    4519 generic.go:356] "Generic (PLEG): container finished" podID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerID="17a5ff0a2e335b895351f610f4cf58e332bf13329ba25e75438adda3a4c801fb" exitCode=60
Oct 31 16:30:03.294487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.294401    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerDied","Data":"17a5ff0a2e335b895351f610f4cf58e332bf13329ba25e75438adda3a4c801fb"}
Oct 31 16:30:03.294717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.294671    4519 scope.go:122] "RemoveContainer" containerID="17a5ff0a2e335b895351f610f4cf58e332bf13329ba25e75438adda3a4c801fb"
Oct 31 16:30:03.296294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.296258    4519 generic.go:356] "Generic (PLEG): container finished" podID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerID="37d1bd10675dded399b529b2dd214f8872de05a3c3b619dcba71a053836691c7" exitCode=60
Oct 31 16:30:03.296342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.296330    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerDied","Data":"37d1bd10675dded399b529b2dd214f8872de05a3c3b619dcba71a053836691c7"}
Oct 31 16:30:03.296553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.296535    4519 scope.go:122] "RemoveContainer" containerID="37d1bd10675dded399b529b2dd214f8872de05a3c3b619dcba71a053836691c7"
Oct 31 16:30:03.298173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.298153    4519 generic.go:356] "Generic (PLEG): container finished" podID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerID="b572f2b160c0b428ec06f5b3d680fd4de662276c1901d354640c032fea041580" exitCode=60
Oct 31 16:30:03.298273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.298236    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerDied","Data":"b572f2b160c0b428ec06f5b3d680fd4de662276c1901d354640c032fea041580"}
Oct 31 16:30:03.298390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.298372    4519 scope.go:122] "RemoveContainer" containerID="b572f2b160c0b428ec06f5b3d680fd4de662276c1901d354640c032fea041580"
Oct 31 16:30:03.299840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.299823    4519 generic.go:356] "Generic (PLEG): container finished" podID="c0801f85-c125-4608-bb37-a49eb4eab201" containerID="2a60c60e8f94941cff573cdfa4b8df01a5c4e6076fc67de4dea5d6cba6172d8c" exitCode=60
Oct 31 16:30:03.299929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.299906    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" event={"ID":"c0801f85-c125-4608-bb37-a49eb4eab201","Type":"ContainerDied","Data":"2a60c60e8f94941cff573cdfa4b8df01a5c4e6076fc67de4dea5d6cba6172d8c"}
Oct 31 16:30:03.300070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.300056    4519 scope.go:122] "RemoveContainer" containerID="2a60c60e8f94941cff573cdfa4b8df01a5c4e6076fc67de4dea5d6cba6172d8c"
Oct 31 16:30:03.301485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.301446    4519 generic.go:356] "Generic (PLEG): container finished" podID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerID="4882d569ce5e6060ac14abd82f0f20d29a4fed62395c3a9ae59620be6ee8703b" exitCode=60
Oct 31 16:30:03.301586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.301523    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerDied","Data":"4882d569ce5e6060ac14abd82f0f20d29a4fed62395c3a9ae59620be6ee8703b"}
Oct 31 16:30:03.301743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:03.301729    4519 scope.go:122] "RemoveContainer" containerID="4882d569ce5e6060ac14abd82f0f20d29a4fed62395c3a9ae59620be6ee8703b"
Oct 31 16:30:04.313111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.313068    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" event={"ID":"c0801f85-c125-4608-bb37-a49eb4eab201","Type":"ContainerStarted","Data":"19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac"}
Oct 31 16:30:04.315113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.315076    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerStarted","Data":"2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b"}
Oct 31 16:30:04.317295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.317265    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerStarted","Data":"7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab"}
Oct 31 16:30:04.319448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.319416    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerStarted","Data":"c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813"}
Oct 31 16:30:04.321694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.321669    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerStarted","Data":"13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343"}
Oct 31 16:30:04.324131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.324109    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerStarted","Data":"1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd"}
Oct 31 16:30:04.325895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.325876    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerStarted","Data":"64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137"}
Oct 31 16:30:04.425054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.425000    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" podStartSLOduration=4.424986119 podStartE2EDuration="4.424986119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:04.419890151 +0000 UTC m=+1648.119819463" watchObservedRunningTime="2026-10-31 16:30:04.424986119 +0000 UTC m=+1648.124915426"
Oct 31 16:30:04.851734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.851626    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" podStartSLOduration=4.8516125500000005 podStartE2EDuration="4.85161255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:04.849944834 +0000 UTC m=+1648.549874145" watchObservedRunningTime="2026-10-31 16:30:04.85161255 +0000 UTC m=+1648.551541860"
Oct 31 16:30:04.965190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:04.965126    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" podStartSLOduration=4.965107714 podStartE2EDuration="4.965107714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:04.961828238 +0000 UTC m=+1648.661757553" watchObservedRunningTime="2026-10-31 16:30:04.965107714 +0000 UTC m=+1648.665037027"
Oct 31 16:30:05.072069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.072018    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" podStartSLOduration=5.072003008 podStartE2EDuration="5.072003008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:05.069717187 +0000 UTC m=+1648.769646499" watchObservedRunningTime="2026-10-31 16:30:05.072003008 +0000 UTC m=+1648.771932319"
Oct 31 16:30:05.337366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.337334    4519 generic.go:356] "Generic (PLEG): container finished" podID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerID="2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b" exitCode=60
Oct 31 16:30:05.337858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.337413    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerDied","Data":"2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b"}
Oct 31 16:30:05.337858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.337517    4519 scope.go:122] "RemoveContainer" containerID="4882d569ce5e6060ac14abd82f0f20d29a4fed62395c3a9ae59620be6ee8703b"
Oct 31 16:30:05.338007 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.337866    4519 scope.go:122] "RemoveContainer" containerID="2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b"
Oct 31 16:30:05.338367 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.338336    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf_odh-ai-gateway-infra(ebc30e44-dfd2-417c-83e2-b0b3c5331259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259"
Oct 31 16:30:05.339751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.339728    4519 generic.go:356] "Generic (PLEG): container finished" podID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerID="7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab" exitCode=60
Oct 31 16:30:05.339864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.339799    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerDied","Data":"7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab"}
Oct 31 16:30:05.340105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.340090    4519 scope.go:122] "RemoveContainer" containerID="7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab"
Oct 31 16:30:05.340516 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.340494    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q_odh-ai-gateway-infra(36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679"
Oct 31 16:30:05.341948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.341928    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerID="c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813" exitCode=60
Oct 31 16:30:05.342032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.341964    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerDied","Data":"c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813"}
Oct 31 16:30:05.342284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.342259    4519 scope.go:122] "RemoveContainer" containerID="c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813"
Oct 31 16:30:05.342647 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.342614    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk_odh-ai-gateway-infra(d5af6878-2c2c-48f8-9753-a6da970432d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0"
Oct 31 16:30:05.344121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.344103    4519 generic.go:356] "Generic (PLEG): container finished" podID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerID="13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343" exitCode=60
Oct 31 16:30:05.344210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.344182    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerDied","Data":"13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343"}
Oct 31 16:30:05.344487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.344442    4519 scope.go:122] "RemoveContainer" containerID="13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343"
Oct 31 16:30:05.344797 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.344775    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs_odh-ai-gateway-infra(959573d1-3e1b-4868-acce-a7048f8e7ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3"
Oct 31 16:30:05.346484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.346436    4519 generic.go:356] "Generic (PLEG): container finished" podID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerID="1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd" exitCode=60
Oct 31 16:30:05.346613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.346583    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerDied","Data":"1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd"}
Oct 31 16:30:05.346863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.346848    4519 scope.go:122] "RemoveContainer" containerID="1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd"
Oct 31 16:30:05.347092 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.347075    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46_odh-ai-gateway-infra(a385b3bc-e028-4510-bdee-5265ce04f4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2"
Oct 31 16:30:05.348905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.348888    4519 generic.go:356] "Generic (PLEG): container finished" podID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerID="64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137" exitCode=60
Oct 31 16:30:05.348994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.348965    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerDied","Data":"64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137"}
Oct 31 16:30:05.349178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.349163    4519 scope.go:122] "RemoveContainer" containerID="64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137"
Oct 31 16:30:05.349441 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.349420    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj_odh-ai-gateway-infra(337a5581-c4a2-4cfb-91ac-349830a1d098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098"
Oct 31 16:30:05.350887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.350865    4519 generic.go:356] "Generic (PLEG): container finished" podID="c0801f85-c125-4608-bb37-a49eb4eab201" containerID="19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac" exitCode=60
Oct 31 16:30:05.350972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.350948    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" event={"ID":"c0801f85-c125-4608-bb37-a49eb4eab201","Type":"ContainerDied","Data":"19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac"}
Oct 31 16:30:05.351154 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.351136    4519 scope.go:122] "RemoveContainer" containerID="19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac"
Oct 31 16:30:05.351400 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:05.351379    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s_odh-ai-gateway-infra(c0801f85-c125-4608-bb37-a49eb4eab201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" podUID="c0801f85-c125-4608-bb37-a49eb4eab201"
Oct 31 16:30:05.357633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.357607    4519 scope.go:122] "RemoveContainer" containerID="dd09f4c8a298d2435c07080b5ce3c962966a02765f090aa65f212bbb94992249"
Oct 31 16:30:05.374484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.374441    4519 scope.go:122] "RemoveContainer" containerID="5f19d1be9d24606f1b7b39b8056fad94be75e06b2622cfa3674c9fc798da8954"
Oct 31 16:30:05.388701 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.388670    4519 scope.go:122] "RemoveContainer" containerID="17a5ff0a2e335b895351f610f4cf58e332bf13329ba25e75438adda3a4c801fb"
Oct 31 16:30:05.403514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.403493    4519 scope.go:122] "RemoveContainer" containerID="37d1bd10675dded399b529b2dd214f8872de05a3c3b619dcba71a053836691c7"
Oct 31 16:30:05.417632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.417609    4519 scope.go:122] "RemoveContainer" containerID="b572f2b160c0b428ec06f5b3d680fd4de662276c1901d354640c032fea041580"
Oct 31 16:30:05.431138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:05.431113    4519 scope.go:122] "RemoveContainer" containerID="2a60c60e8f94941cff573cdfa4b8df01a5c4e6076fc67de4dea5d6cba6172d8c"
Oct 31 16:30:06.365014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.364976    4519 scope.go:122] "RemoveContainer" containerID="2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b"
Oct 31 16:30:06.365711 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.365679    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf_odh-ai-gateway-infra(ebc30e44-dfd2-417c-83e2-b0b3c5331259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259"
Oct 31 16:30:06.367128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.367096    4519 scope.go:122] "RemoveContainer" containerID="7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab"
Oct 31 16:30:06.367846 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.367652    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q_odh-ai-gateway-infra(36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679"
Oct 31 16:30:06.369265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.369243    4519 scope.go:122] "RemoveContainer" containerID="c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813"
Oct 31 16:30:06.369757 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.369719    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk_odh-ai-gateway-infra(d5af6878-2c2c-48f8-9753-a6da970432d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0"
Oct 31 16:30:06.372950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.372932    4519 scope.go:122] "RemoveContainer" containerID="13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343"
Oct 31 16:30:06.373224 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.373206    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs_odh-ai-gateway-infra(959573d1-3e1b-4868-acce-a7048f8e7ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3"
Oct 31 16:30:06.374549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.374528    4519 scope.go:122] "RemoveContainer" containerID="1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd"
Oct 31 16:30:06.374823 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.374801    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46_odh-ai-gateway-infra(a385b3bc-e028-4510-bdee-5265ce04f4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2"
Oct 31 16:30:06.376037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.376020    4519 scope.go:122] "RemoveContainer" containerID="64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137"
Oct 31 16:30:06.376277 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.376255    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj_odh-ai-gateway-infra(337a5581-c4a2-4cfb-91ac-349830a1d098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098"
Oct 31 16:30:06.377662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:06.377645    4519 scope.go:122] "RemoveContainer" containerID="19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac"
Oct 31 16:30:06.377861 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:06.377844    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s_odh-ai-gateway-infra(c0801f85-c125-4608-bb37-a49eb4eab201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" podUID="c0801f85-c125-4608-bb37-a49eb4eab201"
Oct 31 16:30:09.485535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:09.485496    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"]
Oct 31 16:30:09.485963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:09.485823    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy" containerID="cri-o://e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941" gracePeriod=30
Oct 31 16:30:09.533701 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:09.533667    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"]
Oct 31 16:30:09.533919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:09.533896    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy" containerID="cri-o://2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc" gracePeriod=30
Oct 31 16:30:09.538658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:09.538630    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"]
Oct 31 16:30:09.538869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:09.538846    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy" containerID="cri-o://e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0" gracePeriod=30
Oct 31 16:30:12.685220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:12.685187    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:30:14.852707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:14.852671    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:30:14.915344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:14.915319    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:30:14.931030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:14.931009    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:30:15.010369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010333    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.010586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010423    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.010586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010495    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.010586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010524    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.010586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010576    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010666    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010705    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010751    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010780    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010814    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010832    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.010856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010843    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010894    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010953    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.010992    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011028    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011041    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011076    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011107    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011136    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.011291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011253    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011302    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011364    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011432    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011433    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011487    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011545    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011584    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy\") pod \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\" (UID: \"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket\") pod \"01ca8c18-1992-4714-93c3-6861075fdb02\" (UID: \"01ca8c18-1992-4714-93c3-6861075fdb02\") "
Oct 31 16:30:15.011866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.011843    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.012381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.012020    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.012381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.012205    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.012381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.012291    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:15.013136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.012369    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.013136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.012619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq\") pod \"eba0ba18-0be8-4c12-939d-0f3800da54bf\" (UID: \"eba0ba18-0be8-4c12-939d-0f3800da54bf\") "
Oct 31 16:30:15.013136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.012816    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.014079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014042    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014083    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014097    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014112    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ca8c18-1992-4714-93c3-6861075fdb02-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014127    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014141    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014153    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014166    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014179    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014193    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014207    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba0ba18-0be8-4c12-939d-0f3800da54bf-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.014916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.014747    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.016636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.016305    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:15.016636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.016420    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "kube-api-access-5nkmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.017086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.016879    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.017520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.017238    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.018305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.018279    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.019685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.019665    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "kube-api-access-9lznj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.020632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020406    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.020632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020508    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:15.020632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020536    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token" pod "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" (UID: "ddbe4ce9-491e-41ea-bb87-0f09d5ec374a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.020632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020547    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "kube-api-access-7h2cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.020632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:15.020929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020641    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.020929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020758    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token" pod "01ca8c18-1992-4714-93c3-6861075fdb02" (UID: "01ca8c18-1992-4714-93c3-6861075fdb02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.020929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.020759    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.022082 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.022052    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token" pod "eba0ba18-0be8-4c12-939d-0f3800da54bf" (UID: "eba0ba18-0be8-4c12-939d-0f3800da54bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.115830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115795    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ca8c18-1992-4714-93c3-6861075fdb02-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.115830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115830    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115841    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2cq\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-kube-api-access-7h2cq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115851    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkmj\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-kube-api-access-5nkmj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115862    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115870    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lznj\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-kube-api-access-9lznj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115880    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115888    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115897    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115909    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115922    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ca8c18-1992-4714-93c3-6861075fdb02-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115936    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba0ba18-0be8-4c12-939d-0f3800da54bf-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115950    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba0ba18-0be8-4c12-939d-0f3800da54bf-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115965    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115978    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ca8c18-1992-4714-93c3-6861075fdb02-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.116047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.115991    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.432211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.432136    4519 scope.go:122] "RemoveContainer" containerID="c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813"
Oct 31 16:30:15.432364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.432330    4519 scope.go:122] "RemoveContainer" containerID="13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343"
Oct 31 16:30:15.432562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.432542    4519 scope.go:122] "RemoveContainer" containerID="1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd"
Oct 31 16:30:15.463644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.463622    4519 generic.go:356] "Generic (PLEG): container finished" podID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerID="e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941" exitCode=0
Oct 31 16:30:15.463721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.463668    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"
Oct 31 16:30:15.463721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.463707    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" event={"ID":"eba0ba18-0be8-4c12-939d-0f3800da54bf","Type":"ContainerDied","Data":"e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941"}
Oct 31 16:30:15.463806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.463763    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv" event={"ID":"eba0ba18-0be8-4c12-939d-0f3800da54bf","Type":"ContainerDied","Data":"f0aa494f323d2e39fb4ed32f5a1ab0de12a37476bca74a11ef8bb0ffc67e579b"}
Oct 31 16:30:15.463806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.463793    4519 scope.go:122] "RemoveContainer" containerID="e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941"
Oct 31 16:30:15.465335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.465307    4519 generic.go:356] "Generic (PLEG): container finished" podID="01ca8c18-1992-4714-93c3-6861075fdb02" containerID="e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0" exitCode=0
Oct 31 16:30:15.465476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.465338    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"
Oct 31 16:30:15.465476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.465344    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" event={"ID":"01ca8c18-1992-4714-93c3-6861075fdb02","Type":"ContainerDied","Data":"e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0"}
Oct 31 16:30:15.465476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.465398    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k" event={"ID":"01ca8c18-1992-4714-93c3-6861075fdb02","Type":"ContainerDied","Data":"061f52ccfcee28e77360ce782669492f3087f74c047a9b795884b93f697e46d7"}
Oct 31 16:30:15.467191 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.467168    4519 generic.go:356] "Generic (PLEG): container finished" podID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerID="2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc" exitCode=0
Oct 31 16:30:15.467352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.467201    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"
Oct 31 16:30:15.467352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.467244    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" event={"ID":"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a","Type":"ContainerDied","Data":"2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc"}
Oct 31 16:30:15.467352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.467289    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc" event={"ID":"ddbe4ce9-491e-41ea-bb87-0f09d5ec374a","Type":"ContainerDied","Data":"d32706adbb7012ffbbe06f9c59e7eb4a0498f7fa4a2554d022d29afc70072aaf"}
Oct 31 16:30:15.491070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.491040    4519 scope.go:122] "RemoveContainer" containerID="e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941"
Oct 31 16:30:15.491623 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:15.491576    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941\": container with ID starting with e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941 not found: ID does not exist" containerID="e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941"
Oct 31 16:30:15.491737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.491620    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941"} err="failed to get container status \"e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941\": rpc error: code = NotFound desc = could not find container \"e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941\": container with ID starting with e3831d76377e677f64be9a19dbad000a7746dd8bc1a77d280c5e0ad664528941 not found: ID does not exist"
Oct 31 16:30:15.491737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.491645    4519 scope.go:122] "RemoveContainer" containerID="e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0"
Oct 31 16:30:15.505982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.505960    4519 scope.go:122] "RemoveContainer" containerID="e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0"
Oct 31 16:30:15.506323 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:15.506297    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0\": container with ID starting with e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0 not found: ID does not exist" containerID="e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0"
Oct 31 16:30:15.506430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.506335    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0"} err="failed to get container status \"e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0\": rpc error: code = NotFound desc = could not find container \"e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0\": container with ID starting with e174c95cea300ed5b354684564b0cc8e8e2eb802b40fd5c17068e0210cb540e0 not found: ID does not exist"
Oct 31 16:30:15.506430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.506358    4519 scope.go:122] "RemoveContainer" containerID="2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc"
Oct 31 16:30:15.540714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.540688    4519 scope.go:122] "RemoveContainer" containerID="2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc"
Oct 31 16:30:15.541035 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:15.541016    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc\": container with ID starting with 2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc not found: ID does not exist" containerID="2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc"
Oct 31 16:30:15.541081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.541043    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc"} err="failed to get container status \"2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc\": rpc error: code = NotFound desc = could not find container \"2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc\": container with ID starting with 2927cf18088feec3c6877e67bc404dd22587f11468f1f50a9be3ecb0f89f02fc not found: ID does not exist"
Oct 31 16:30:15.624342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.624311    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"]
Oct 31 16:30:15.634151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.634118    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-99qrc"]
Oct 31 16:30:15.736716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.736661    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"]
Oct 31 16:30:15.743727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.743698    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-cln4k"]
Oct 31 16:30:15.843243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.843214    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"]
Oct 31 16:30:15.850383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:15.850354    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-dmssv"]
Oct 31 16:30:16.440892 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.440861    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" path="/var/lib/kubelet/pods/01ca8c18-1992-4714-93c3-6861075fdb02/volumes"
Oct 31 16:30:16.441349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.441333    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" path="/var/lib/kubelet/pods/ddbe4ce9-491e-41ea-bb87-0f09d5ec374a/volumes"
Oct 31 16:30:16.441790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.441777    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" path="/var/lib/kubelet/pods/eba0ba18-0be8-4c12-939d-0f3800da54bf/volumes"
Oct 31 16:30:16.442876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.442846    4519 scope.go:122] "RemoveContainer" containerID="2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b"
Oct 31 16:30:16.443026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.443000    4519 scope.go:122] "RemoveContainer" containerID="7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab"
Oct 31 16:30:16.443158 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.443138    4519 scope.go:122] "RemoveContainer" containerID="64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137"
Oct 31 16:30:16.482059 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.482029    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerStarted","Data":"c3e785aa9a6629d7f2aed6aead35bad7eea2a5f7e61c93bf9628025d0ce91ce6"}
Oct 31 16:30:16.484231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.484208    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerStarted","Data":"92621f8a03f83bd58e477001fa9482716d71faae68f3d9279f7691154b8eb758"}
Oct 31 16:30:16.484680 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.484661    4519 scope.go:122] "RemoveContainer" containerID="92621f8a03f83bd58e477001fa9482716d71faae68f3d9279f7691154b8eb758"
Oct 31 16:30:16.485064 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:16.485037    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs_odh-ai-gateway-infra(959573d1-3e1b-4868-acce-a7048f8e7ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3"
Oct 31 16:30:16.486518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:16.486496    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerStarted","Data":"153863bba26b7930e3b16ff0c25eef6f003efd45a186e41ec328838b43369d1c"}
Oct 31 16:30:17.499172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.499135    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerStarted","Data":"bbf5f658bc921c637b19e87d33bac2638eb338b395116606785b233b9ef5c67c"}
Oct 31 16:30:17.501367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.501339    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerStarted","Data":"f13d88577dca2b63fd8d4c11d6e786d509504812d046891aad7be1db93086d40"}
Oct 31 16:30:17.503314 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.503293    4519 generic.go:356] "Generic (PLEG): container finished" podID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerID="c3e785aa9a6629d7f2aed6aead35bad7eea2a5f7e61c93bf9628025d0ce91ce6" exitCode=60
Oct 31 16:30:17.503433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.503366    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerDied","Data":"c3e785aa9a6629d7f2aed6aead35bad7eea2a5f7e61c93bf9628025d0ce91ce6"}
Oct 31 16:30:17.503433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.503422    4519 scope.go:122] "RemoveContainer" containerID="c76e036f1fd1b4c6a1a28458c068ef300da71bf1f7c8a8c0b2b8a56ab18d2813"
Oct 31 16:30:17.503858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.503841    4519 scope.go:122] "RemoveContainer" containerID="c3e785aa9a6629d7f2aed6aead35bad7eea2a5f7e61c93bf9628025d0ce91ce6"
Oct 31 16:30:17.504425 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:17.504312    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk_odh-ai-gateway-infra(d5af6878-2c2c-48f8-9753-a6da970432d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0"
Oct 31 16:30:17.505511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.505491    4519 generic.go:356] "Generic (PLEG): container finished" podID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerID="92621f8a03f83bd58e477001fa9482716d71faae68f3d9279f7691154b8eb758" exitCode=60
Oct 31 16:30:17.505616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.505567    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerDied","Data":"92621f8a03f83bd58e477001fa9482716d71faae68f3d9279f7691154b8eb758"}
Oct 31 16:30:17.505923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.505907    4519 scope.go:122] "RemoveContainer" containerID="92621f8a03f83bd58e477001fa9482716d71faae68f3d9279f7691154b8eb758"
Oct 31 16:30:17.506264 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:17.506231    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs_odh-ai-gateway-infra(959573d1-3e1b-4868-acce-a7048f8e7ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3"
Oct 31 16:30:17.507544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.507519    4519 generic.go:356] "Generic (PLEG): container finished" podID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerID="153863bba26b7930e3b16ff0c25eef6f003efd45a186e41ec328838b43369d1c" exitCode=60
Oct 31 16:30:17.507631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.507556    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerDied","Data":"153863bba26b7930e3b16ff0c25eef6f003efd45a186e41ec328838b43369d1c"}
Oct 31 16:30:17.507908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.507892    4519 scope.go:122] "RemoveContainer" containerID="153863bba26b7930e3b16ff0c25eef6f003efd45a186e41ec328838b43369d1c"
Oct 31 16:30:17.508152 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:17.508130    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46_odh-ai-gateway-infra(a385b3bc-e028-4510-bdee-5265ce04f4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2"
Oct 31 16:30:17.509567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.509539    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerStarted","Data":"7123da0e45a6223ea78ee60a85ec77b785012af964a5440c2d85708107929dd0"}
Oct 31 16:30:17.518146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.518126    4519 scope.go:122] "RemoveContainer" containerID="13419ee8a2b1ca275a4783e7140a4a0f6c8356a87296832645d679f09cf2d343"
Oct 31 16:30:17.532213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.532186    4519 scope.go:122] "RemoveContainer" containerID="1de8fdfe9e2ec77d91ba8925d1c79c3694ab7b4d9e23933464b75d49356b1acd"
Oct 31 16:30:17.603285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.603258    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"]
Oct 31 16:30:17.708268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.708238    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"]
Oct 31 16:30:17.815585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:17.815555    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"]
Oct 31 16:30:18.441315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.441285    4519 scope.go:122] "RemoveContainer" containerID="19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac"
Oct 31 16:30:18.519607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.519577    4519 generic.go:356] "Generic (PLEG): container finished" podID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerID="bbf5f658bc921c637b19e87d33bac2638eb338b395116606785b233b9ef5c67c" exitCode=60
Oct 31 16:30:18.520022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.519646    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerDied","Data":"bbf5f658bc921c637b19e87d33bac2638eb338b395116606785b233b9ef5c67c"}
Oct 31 16:30:18.520022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.519702    4519 scope.go:122] "RemoveContainer" containerID="2fca1ad2a7be71f8b1f666cb26e9c7cafa14efb80c4bbf4df921c65543a38b6b"
Oct 31 16:30:18.520151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.520131    4519 scope.go:122] "RemoveContainer" containerID="bbf5f658bc921c637b19e87d33bac2638eb338b395116606785b233b9ef5c67c"
Oct 31 16:30:18.520552 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:18.520531    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf_odh-ai-gateway-infra(ebc30e44-dfd2-417c-83e2-b0b3c5331259)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259"
Oct 31 16:30:18.521852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.521831    4519 generic.go:356] "Generic (PLEG): container finished" podID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerID="f13d88577dca2b63fd8d4c11d6e786d509504812d046891aad7be1db93086d40" exitCode=60
Oct 31 16:30:18.521943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.521903    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerDied","Data":"f13d88577dca2b63fd8d4c11d6e786d509504812d046891aad7be1db93086d40"}
Oct 31 16:30:18.522341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.522319    4519 scope.go:122] "RemoveContainer" containerID="f13d88577dca2b63fd8d4c11d6e786d509504812d046891aad7be1db93086d40"
Oct 31 16:30:18.522712 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:18.522692    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q_odh-ai-gateway-infra(36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679"
Oct 31 16:30:18.529714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.529687    4519 generic.go:356] "Generic (PLEG): container finished" podID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerID="7123da0e45a6223ea78ee60a85ec77b785012af964a5440c2d85708107929dd0" exitCode=60
Oct 31 16:30:18.529822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.529727    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerDied","Data":"7123da0e45a6223ea78ee60a85ec77b785012af964a5440c2d85708107929dd0"}
Oct 31 16:30:18.530124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.530106    4519 scope.go:122] "RemoveContainer" containerID="7123da0e45a6223ea78ee60a85ec77b785012af964a5440c2d85708107929dd0"
Oct 31 16:30:18.530515 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:18.530490    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj_odh-ai-gateway-infra(337a5581-c4a2-4cfb-91ac-349830a1d098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098"
Oct 31 16:30:18.538915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.538832    4519 scope.go:122] "RemoveContainer" containerID="7d17ea44d63869086beef21b3f0e7ed3943ccf0bc9ac95288d62b8d18e82abab"
Oct 31 16:30:18.557031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.557009    4519 scope.go:122] "RemoveContainer" containerID="64ec984b5cea173dff9c74293142faf430c07b3c62e568731a58e3ec3b6f4137"
Oct 31 16:30:18.629018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.628914    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"]
Oct 31 16:30:18.728248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.728194    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"]
Oct 31 16:30:18.754169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.754140    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:18.775528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.775493    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm\" (UniqueName: \"kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm\") pod \"d5af6878-2c2c-48f8-9753-a6da970432d0\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") "
Oct 31 16:30:18.775720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.775558    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca\") pod \"d5af6878-2c2c-48f8-9753-a6da970432d0\" (UID: \"d5af6878-2c2c-48f8-9753-a6da970432d0\") "
Oct 31 16:30:18.776725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.776701    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca" pod "d5af6878-2c2c-48f8-9753-a6da970432d0" (UID: "d5af6878-2c2c-48f8-9753-a6da970432d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:18.866225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.866186    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm" pod "d5af6878-2c2c-48f8-9753-a6da970432d0" (UID: "d5af6878-2c2c-48f8-9753-a6da970432d0"). InnerVolumeSpecName "kube-api-access-jq4cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:18.878351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.878327    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4cm\" (UniqueName: \"kubernetes.io/projected/d5af6878-2c2c-48f8-9753-a6da970432d0-kube-api-access-jq4cm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:18.878964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.878353    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5af6878-2c2c-48f8-9753-a6da970432d0-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:18.917262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.917217    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:18.923103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.923069    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:18.979593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.979565    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv\" (UniqueName: \"kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv\") pod \"a385b3bc-e028-4510-bdee-5265ce04f4a2\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") "
Oct 31 16:30:18.979806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.979642    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv\" (UniqueName: \"kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv\") pod \"959573d1-3e1b-4868-acce-a7048f8e7ec3\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") "
Oct 31 16:30:18.979894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.979818    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca\") pod \"959573d1-3e1b-4868-acce-a7048f8e7ec3\" (UID: \"959573d1-3e1b-4868-acce-a7048f8e7ec3\") "
Oct 31 16:30:18.979894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.979850    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca\") pod \"a385b3bc-e028-4510-bdee-5265ce04f4a2\" (UID: \"a385b3bc-e028-4510-bdee-5265ce04f4a2\") "
Oct 31 16:30:18.980206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.980174    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca" pod "959573d1-3e1b-4868-acce-a7048f8e7ec3" (UID: "959573d1-3e1b-4868-acce-a7048f8e7ec3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:18.980344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.980217    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca" pod "a385b3bc-e028-4510-bdee-5265ce04f4a2" (UID: "a385b3bc-e028-4510-bdee-5265ce04f4a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:18.980398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.980382    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/959573d1-3e1b-4868-acce-a7048f8e7ec3-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:18.980438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.980402    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a385b3bc-e028-4510-bdee-5265ce04f4a2-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:18.981781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.981752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv" pod "a385b3bc-e028-4510-bdee-5265ce04f4a2" (UID: "a385b3bc-e028-4510-bdee-5265ce04f4a2"). InnerVolumeSpecName "kube-api-access-mv4lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:18.981900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:18.981849    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv" pod "959573d1-3e1b-4868-acce-a7048f8e7ec3" (UID: "959573d1-3e1b-4868-acce-a7048f8e7ec3"). InnerVolumeSpecName "kube-api-access-g8gtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:19.046959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.046925    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"]
Oct 31 16:30:19.081404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.081371    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4lv\" (UniqueName: \"kubernetes.io/projected/a385b3bc-e028-4510-bdee-5265ce04f4a2-kube-api-access-mv4lv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.081404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.081401    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8gtv\" (UniqueName: \"kubernetes.io/projected/959573d1-3e1b-4868-acce-a7048f8e7ec3-kube-api-access-g8gtv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.541733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.541698    4519 generic.go:356] "Generic (PLEG): container finished" podID="c0801f85-c125-4608-bb37-a49eb4eab201" containerID="392676df4f09a284ca686c65cebd8f8e05c6310eae5e0b359dfd0172bf261281" exitCode=60
Oct 31 16:30:19.542285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.541783    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" event={"ID":"c0801f85-c125-4608-bb37-a49eb4eab201","Type":"ContainerDied","Data":"392676df4f09a284ca686c65cebd8f8e05c6310eae5e0b359dfd0172bf261281"}
Oct 31 16:30:19.542285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.541840    4519 scope.go:122] "RemoveContainer" containerID="19b72902edaebe162d96a2f9a9123123a05cfa56bad0a6e22dc0ffc4037ef5ac"
Oct 31 16:30:19.542428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.542321    4519 scope.go:122] "RemoveContainer" containerID="392676df4f09a284ca686c65cebd8f8e05c6310eae5e0b359dfd0172bf261281"
Oct 31 16:30:19.542836 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:19.542806    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s_odh-ai-gateway-infra(c0801f85-c125-4608-bb37-a49eb4eab201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" podUID="c0801f85-c125-4608-bb37-a49eb4eab201"
Oct 31 16:30:19.547445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.547425    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"
Oct 31 16:30:19.547606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.547442    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk" event={"ID":"d5af6878-2c2c-48f8-9753-a6da970432d0","Type":"ContainerDied","Data":"9b62c08d1ed4c11f997ce5509a69fc81b8452eb0e95df090b3e20d3bd2a99349"}
Oct 31 16:30:19.549309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.549293    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"
Oct 31 16:30:19.549811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.549295    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs" event={"ID":"959573d1-3e1b-4868-acce-a7048f8e7ec3","Type":"ContainerDied","Data":"cbde54b608a484d61831fbf5b88a3c963ac39b46b9a7d6ba28508b1d9725a20a"}
Oct 31 16:30:19.551314 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.551294    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"
Oct 31 16:30:19.551413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.551359    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46" event={"ID":"a385b3bc-e028-4510-bdee-5265ce04f4a2","Type":"ContainerDied","Data":"a95ad963d09c53a922a0e4aa4af47d0e3ec22bc2e0abd29e8ec80df6f3f1214b"}
Oct 31 16:30:19.564045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.564022    4519 scope.go:122] "RemoveContainer" containerID="c3e785aa9a6629d7f2aed6aead35bad7eea2a5f7e61c93bf9628025d0ce91ce6"
Oct 31 16:30:19.583991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.583612    4519 scope.go:122] "RemoveContainer" containerID="92621f8a03f83bd58e477001fa9482716d71faae68f3d9279f7691154b8eb758"
Oct 31 16:30:19.604429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.603716    4519 scope.go:122] "RemoveContainer" containerID="153863bba26b7930e3b16ff0c25eef6f003efd45a186e41ec328838b43369d1c"
Oct 31 16:30:19.764073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.763999    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:19.793185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.793153    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca\") pod \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") "
Oct 31 16:30:19.793877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.793852    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs\" (UniqueName: \"kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs\") pod \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\" (UID: \"ebc30e44-dfd2-417c-83e2-b0b3c5331259\") "
Oct 31 16:30:19.795145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.793737    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca" pod "ebc30e44-dfd2-417c-83e2-b0b3c5331259" (UID: "ebc30e44-dfd2-417c-83e2-b0b3c5331259"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:19.796348 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.796327    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc30e44-dfd2-417c-83e2-b0b3c5331259-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.799561 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.799528    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs" pod "ebc30e44-dfd2-417c-83e2-b0b3c5331259" (UID: "ebc30e44-dfd2-417c-83e2-b0b3c5331259"). InnerVolumeSpecName "kube-api-access-nhmvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:19.836613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.836588    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:19.840239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.840217    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:19.869473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.869432    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"]
Oct 31 16:30:19.874686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.874653    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a1b30a15-29891070-f2drk"]
Oct 31 16:30:19.897101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.897070    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca\") pod \"337a5581-c4a2-4cfb-91ac-349830a1d098\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") "
Oct 31 16:30:19.897269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.897250    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf\" (UniqueName: \"kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf\") pod \"337a5581-c4a2-4cfb-91ac-349830a1d098\" (UID: \"337a5581-c4a2-4cfb-91ac-349830a1d098\") "
Oct 31 16:30:19.897466 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.897435    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca" pod "337a5581-c4a2-4cfb-91ac-349830a1d098" (UID: "337a5581-c4a2-4cfb-91ac-349830a1d098"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:19.897528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.897505    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca\") pod \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") "
Oct 31 16:30:19.897578 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.897545    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n\" (UniqueName: \"kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n\") pod \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\" (UID: \"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679\") "
Oct 31 16:30:19.897846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.897820    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca" pod "36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" (UID: "36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:19.898217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.898200    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/337a5581-c4a2-4cfb-91ac-349830a1d098-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.898261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.898220    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhmvs\" (UniqueName: \"kubernetes.io/projected/ebc30e44-dfd2-417c-83e2-b0b3c5331259-kube-api-access-nhmvs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.898261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.898230    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.899360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.899341    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf" pod "337a5581-c4a2-4cfb-91ac-349830a1d098" (UID: "337a5581-c4a2-4cfb-91ac-349830a1d098"). InnerVolumeSpecName "kube-api-access-7dtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:19.899586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.899570    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n" pod "36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" (UID: "36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679"). InnerVolumeSpecName "kube-api-access-65t7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:19.999694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.999596    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65t7n\" (UniqueName: \"kubernetes.io/projected/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679-kube-api-access-65t7n\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.999694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:19.999629    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtmf\" (UniqueName: \"kubernetes.io/projected/337a5581-c4a2-4cfb-91ac-349830a1d098-kube-api-access-7dtmf\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:20.197223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.197191    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"]
Oct 31 16:30:20.203954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.203919    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891070-kkc46"]
Oct 31 16:30:20.302172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.302082    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"]
Oct 31 16:30:20.306885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.306855    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891070-hmwfs"]
Oct 31 16:30:20.447663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.447630    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" path="/var/lib/kubelet/pods/959573d1-3e1b-4868-acce-a7048f8e7ec3/volumes"
Oct 31 16:30:20.448176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.448157    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" path="/var/lib/kubelet/pods/a385b3bc-e028-4510-bdee-5265ce04f4a2/volumes"
Oct 31 16:30:20.448616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.448599    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" path="/var/lib/kubelet/pods/d5af6878-2c2c-48f8-9753-a6da970432d0/volumes"
Oct 31 16:30:20.587611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.587497    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf" event={"ID":"ebc30e44-dfd2-417c-83e2-b0b3c5331259","Type":"ContainerDied","Data":"3258c9db4ef00bac73b09a4475ffb23189a3a6099dd931daf54536f3f9d44b4e"}
Oct 31 16:30:20.587611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.587578    4519 scope.go:122] "RemoveContainer" containerID="bbf5f658bc921c637b19e87d33bac2638eb338b395116606785b233b9ef5c67c"
Oct 31 16:30:20.588032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.587738    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"
Oct 31 16:30:20.607004 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.606964    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q" event={"ID":"36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679","Type":"ContainerDied","Data":"8f5bc262e3afffcf11297b3fc978d5e31c8406ce5d63848dc1d6b9ad944d6e1c"}
Oct 31 16:30:20.607327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.607301    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"
Oct 31 16:30:20.622522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.622481    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj" event={"ID":"337a5581-c4a2-4cfb-91ac-349830a1d098","Type":"ContainerDied","Data":"03bab21f3abaaef030325d8e40af3c44ec82922605c709c4234da81b19a7c3f0"}
Oct 31 16:30:20.622709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.622613    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"
Oct 31 16:30:20.643679 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.643646    4519 scope.go:122] "RemoveContainer" containerID="f13d88577dca2b63fd8d4c11d6e786d509504812d046891aad7be1db93086d40"
Oct 31 16:30:20.667378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.667352    4519 scope.go:122] "RemoveContainer" containerID="7123da0e45a6223ea78ee60a85ec77b785012af964a5440c2d85708107929dd0"
Oct 31 16:30:20.675510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.675485    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"]
Oct 31 16:30:20.749975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.749947    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"]
Oct 31 16:30:20.754908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.754883    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ac1273-29891070-rsz4q"]
Oct 31 16:30:20.849291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.849223    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:20.858211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.858182    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"]
Oct 31 16:30:20.862408 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.862384    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ef6d93-29891070-6wwrf"]
Oct 31 16:30:20.921856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.921823    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn\" (UniqueName: \"kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn\") pod \"c0801f85-c125-4608-bb37-a49eb4eab201\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") "
Oct 31 16:30:20.922147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.922125    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca\") pod \"c0801f85-c125-4608-bb37-a49eb4eab201\" (UID: \"c0801f85-c125-4608-bb37-a49eb4eab201\") "
Oct 31 16:30:20.922487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.922447    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca" pod "c0801f85-c125-4608-bb37-a49eb4eab201" (UID: "c0801f85-c125-4608-bb37-a49eb4eab201"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:20.922757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.922740    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801f85-c125-4608-bb37-a49eb4eab201-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:20.923991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.923962    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn" pod "c0801f85-c125-4608-bb37-a49eb4eab201" (UID: "c0801f85-c125-4608-bb37-a49eb4eab201"). InnerVolumeSpecName "kube-api-access-lfqnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:20.963442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.963410    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"]
Oct 31 16:30:20.969243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:20.969217    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-a7907d70-29891070-rhxvj"]
Oct 31 16:30:21.024134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:21.024101    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfqnn\" (UniqueName: \"kubernetes.io/projected/c0801f85-c125-4608-bb37-a49eb4eab201-kube-api-access-lfqnn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:21.637771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:21.637741    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s" event={"ID":"c0801f85-c125-4608-bb37-a49eb4eab201","Type":"ContainerDied","Data":"cdbf0f48ff365d31f6f64d0cfe0d673841b33799b53814b7ed17ec1467fea566"}
Oct 31 16:30:21.638225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:21.637801    4519 scope.go:122] "RemoveContainer" containerID="392676df4f09a284ca686c65cebd8f8e05c6310eae5e0b359dfd0172bf261281"
Oct 31 16:30:21.638225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:21.637749    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"
Oct 31 16:30:21.795017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:21.794974    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"]
Oct 31 16:30:21.798934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:21.798909    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891070-sf95s"]
Oct 31 16:30:22.441650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:22.441617    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" path="/var/lib/kubelet/pods/337a5581-c4a2-4cfb-91ac-349830a1d098/volumes"
Oct 31 16:30:22.442194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:22.442178    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" path="/var/lib/kubelet/pods/36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679/volumes"
Oct 31 16:30:22.442666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:22.442650    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" path="/var/lib/kubelet/pods/c0801f85-c125-4608-bb37-a49eb4eab201/volumes"
Oct 31 16:30:22.443400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:22.443385    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" path="/var/lib/kubelet/pods/ebc30e44-dfd2-417c-83e2-b0b3c5331259/volumes"
Oct 31 16:30:24.006250 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:24.006215    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:30:26.910954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:26.910913    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"]
Oct 31 16:30:26.915060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:26.915027    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a7907d70-vprwp"]
Oct 31 16:30:28.440788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:28.440750    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fa7860-cd5b-489a-97cf-0c7da28c22b0" path="/var/lib/kubelet/pods/40fa7860-cd5b-489a-97cf-0c7da28c22b0/volumes"
Oct 31 16:30:33.480188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:33.480151    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"]
Oct 31 16:30:33.480635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:33.480438    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy" containerID="cri-o://5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06" gracePeriod=30
Oct 31 16:30:38.029742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.029707    4519 scope.go:122] "RemoveContainer" containerID="cab09241317d9f435d4d9668dcdfb43b84a5eb6a7a711874aa7471307b8ae744"
Oct 31 16:30:38.044352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.044321    4519 scope.go:122] "RemoveContainer" containerID="acb667751afa3886f94bc0daf94b5c23e90b012d74219300234ffbaa514afb2a"
Oct 31 16:30:38.057065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.057033    4519 scope.go:122] "RemoveContainer" containerID="d9c8bd51b3120a09e12752c6629046ba1cc50aa2552f70726e96e0456ee493d4"
Oct 31 16:30:38.068231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.068211    4519 scope.go:122] "RemoveContainer" containerID="ba15e4146ae344793688cfb99d589edf83dc1d4c0a2510419e7afda14987c50b"
Oct 31 16:30:38.780901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.780876    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:30:38.878995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.878960    4519 generic.go:356] "Generic (PLEG): container finished" podID="821daccc-0a90-4511-9c5e-613589b2681c" containerID="5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06" exitCode=0
Oct 31 16:30:38.879161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.879014    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"
Oct 31 16:30:38.879161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.879038    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" event={"ID":"821daccc-0a90-4511-9c5e-613589b2681c","Type":"ContainerDied","Data":"5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06"}
Oct 31 16:30:38.879161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.879093    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7" event={"ID":"821daccc-0a90-4511-9c5e-613589b2681c","Type":"ContainerDied","Data":"add3171b3e48c4fa763427c0bee0846f59ea6dd192b8b2b6f27db496b398602a"}
Oct 31 16:30:38.879161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.879129    4519 scope.go:122] "RemoveContainer" containerID="5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06"
Oct 31 16:30:38.889349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.889332    4519 scope.go:122] "RemoveContainer" containerID="2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"
Oct 31 16:30:38.891878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.891855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.891969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.891953    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.891994    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892068 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892053    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892114 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892076    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:38.892114 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892201    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892244    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:38.892403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892383    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892591    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892656    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data\") pod \"821daccc-0a90-4511-9c5e-613589b2681c\" (UID: \"821daccc-0a90-4511-9c5e-613589b2681c\") "
Oct 31 16:30:38.892754 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892734    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:38.892996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.892973    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:38.893228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.893205    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:38.893551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.893519    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821daccc-0a90-4511-9c5e-613589b2681c-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.893551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.893548    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.893697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.893563    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.893697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.893578    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.893697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.893592    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.894336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.894314    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:38.894647 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.894626    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:38.894750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.894680    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:38.894826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.894812    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85" pod "821daccc-0a90-4511-9c5e-613589b2681c" (UID: "821daccc-0a90-4511-9c5e-613589b2681c"). InnerVolumeSpecName "kube-api-access-4lp85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:38.899593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.899576    4519 scope.go:122] "RemoveContainer" containerID="5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06"
Oct 31 16:30:38.899881 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:38.899863    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06\": container with ID starting with 5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06 not found: ID does not exist" containerID="5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06"
Oct 31 16:30:38.899936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.899888    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06"} err="failed to get container status \"5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06\": rpc error: code = NotFound desc = could not find container \"5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06\": container with ID starting with 5b5bb27f9644c6b9cc8ab4e52366a52c5a483ed5da775aff1d6dd822d9667f06 not found: ID does not exist"
Oct 31 16:30:38.899936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.899907    4519 scope.go:122] "RemoveContainer" containerID="2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"
Oct 31 16:30:38.900176 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:38.900152    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c\": container with ID starting with 2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c not found: ID does not exist" containerID="2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"
Oct 31 16:30:38.900243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.900185    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c"} err="failed to get container status \"2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c\": rpc error: code = NotFound desc = could not find container \"2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c\": container with ID starting with 2411f3220ab1849ba90d912bbac67d6161c6edd55a8864f2d826eb5bcbc44f7c not found: ID does not exist"
Oct 31 16:30:38.996487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.996050    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.996487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.996087    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821daccc-0a90-4511-9c5e-613589b2681c-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.996487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.996103    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lp85\" (UniqueName: \"kubernetes.io/projected/821daccc-0a90-4511-9c5e-613589b2681c-kube-api-access-4lp85\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:38.996487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:38.996124    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821daccc-0a90-4511-9c5e-613589b2681c-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:39.369364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:39.369333    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"]
Oct 31 16:30:39.371515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:39.371487    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a7907d70-openshift-default-6d96ff57bf-n52c7"]
Oct 31 16:30:40.441593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:40.441563    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821daccc-0a90-4511-9c5e-613589b2681c" path="/var/lib/kubelet/pods/821daccc-0a90-4511-9c5e-613589b2681c/volumes"
Oct 31 16:30:41.049955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:41.049927    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"]
Oct 31 16:30:41.097170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:41.097140    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20212d6c-zmmmh"]
Oct 31 16:30:42.450367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:30:42.450328    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0920931b-99c7-4ae9-9a9c-c3cef0dc8f32" path="/var/lib/kubelet/pods/0920931b-99c7-4ae9-9a9c-c3cef0dc8f32/volumes"
Oct 31 16:30:54.005743 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:30:54.005707    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:31:22.483954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:22.483915    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:31:24.026618 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:31:24.026585    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:31:34.213826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.213792    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:34.214304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.213849    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:34.403522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.403487    4519 generic.go:356] "Generic (PLEG): container finished" podID="f38d95c8-7437-4a33-8270-1684c798cd86" containerID="7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4" exitCode=0
Oct 31 16:31:34.403727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.403562    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" event={"ID":"f38d95c8-7437-4a33-8270-1684c798cd86","Type":"ContainerDied","Data":"7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"}
Oct 31 16:31:34.404112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.404091    4519 scope.go:122] "RemoveContainer" containerID="7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"
Oct 31 16:31:34.405579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.405557    4519 generic.go:356] "Generic (PLEG): container finished" podID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerID="d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f" exitCode=0
Oct 31 16:31:34.405671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.405594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" event={"ID":"b5a758d2-3317-4ee9-bb46-39e7c5476698","Type":"ContainerDied","Data":"d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"}
Oct 31 16:31:34.405972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.405958    4519 scope.go:122] "RemoveContainer" containerID="d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"
Oct 31 16:31:34.406195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.406166    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.406257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.406243    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.406317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.406304    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.407584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.407559    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.407661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.407622    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.407661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:34.407656    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:35.222274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:35.222237    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:31:35.427078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:35.427037    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" event={"ID":"b5a758d2-3317-4ee9-bb46-39e7c5476698","Type":"ContainerStarted","Data":"ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af"}
Oct 31 16:31:35.429372 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:35.429342    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" event={"ID":"f38d95c8-7437-4a33-8270-1684c798cd86","Type":"ContainerStarted","Data":"b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89"}
Oct 31 16:31:36.005480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:35.992867    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:31:36.005480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:35.993013    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:36.005480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:35.993066    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:36.018987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:36.018947    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:31:36.020025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:36.020003    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 16:31:36.020197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:36.020180    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 16:31:36.992890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:36.992848    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:36.993553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:36.993518    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:37.017913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:37.017878    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 16:31:37.021804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:37.021565    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 16:31:37.994885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:37.994844    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:37.998560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:37.998519    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:38.019741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:38.019705    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 16:31:38.022188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:38.022060    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 16:31:38.191259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:38.191222    4519 scope.go:122] "RemoveContainer" containerID="a091982ff84ac56bfe1a60a7cf7d72ec41f0708379833fd89ee48ea64f2f2f2f"
Oct 31 16:31:38.236784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:38.236759    4519 scope.go:122] "RemoveContainer" containerID="3b24dfd2e7e78994f08431887782c27874e265fbebaeb9a07fe924afa6cadfc8"
Oct 31 16:31:38.997477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:38.997276    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:38.997477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:38.997358    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:39.019875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:39.019842    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 16:31:39.020117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:39.020095    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 16:31:39.994479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:39.993865    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:39.994479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:39.993945    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:40.019349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:40.019318    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 16:31:40.019973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:40.019947    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 16:31:40.991916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:40.991872    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 16:31:40.992215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:40.992185    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 16:31:41.018166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:41.018124    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 16:31:41.018351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:41.018194    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 16:31:41.996683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:41.996649    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:31:41.997182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:41.997151    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:31:41.997971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:41.997951    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:31:42.022933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:42.022902    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:31:42.023304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:42.023277    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:31:42.024238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:42.024216    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:31:50.081904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.081826    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"]
Oct 31 16:31:50.082617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082592    4519 cpu_manager.go:414] "containerMap: removing container" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:50.082617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082615    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082627    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082632    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082646    4519 cpu_manager.go:414] "containerMap: removing container" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082652    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082666    4519 cpu_manager.go:414] "containerMap: removing container" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082670    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082681    4519 cpu_manager.go:414] "containerMap: removing container" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082686    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082692    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082696    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082708    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082714    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082721    4519 cpu_manager.go:414] "containerMap: removing container" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082725    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082731    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082735    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082746    4519 cpu_manager.go:414] "containerMap: removing container" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082751    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082758    4519 cpu_manager.go:414] "containerMap: removing container" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:50.082764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082762    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082776    4519 cpu_manager.go:414] "containerMap: removing container" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082781    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082789    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082793    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082800    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082805    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082811    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082815    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082821    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082825    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082833    4519 cpu_manager.go:414] "containerMap: removing container" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082837    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082850    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082854    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082863    4519 cpu_manager.go:414] "containerMap: removing container" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082867    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082874    4519 cpu_manager.go:414] "containerMap: removing container" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082879    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082884    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082888    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082896    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.082901    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083012    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083022    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083028    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083034    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083039    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083045    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083050    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083054    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbe4ce9-491e-41ea-bb87-0f09d5ec374a" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083059    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083064    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083070    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="959573d1-3e1b-4868-acce-a7048f8e7ec3" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083078    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba0ba18-0be8-4c12-939d-0f3800da54bf" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083084    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083089    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083095    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083101    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083108    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ca8c18-1992-4714-93c3-6861075fdb02" containerName="istio-proxy"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083115    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083120    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.083378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.083127    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:50.087803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.087783    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.138284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.138255    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-487213-openshift-default-dockercfg-4xk7z\""
Oct 31 16:31:50.190121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.190087    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"]
Oct 31 16:31:50.214888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.214841    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.214908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.214944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.214989    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.215055    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzz6\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215298 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.215130    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215298 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.215172    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.215337    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.215396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.215383    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.325916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.325875    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.326624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.326600    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.326993    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.327175    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.327396    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.327557    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.327693    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzz6\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.327842    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.327974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.327961    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.328932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.328911    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.329065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.326530    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.329718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.329696    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.331220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.330081    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.331346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.331159    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.372189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.372075    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.372924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.372896    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.397536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.397471    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzz6\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.400547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.400405    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token\") pod \"e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") " pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:50.401182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:50.400956    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:51.109692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.109658    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"]
Oct 31 16:31:51.114727 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:31:51.114700    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d85ad72_dbb1_4cf1_bd4d_82746fac3eb3.slice/crio-05bbe83caea16fae2910005b2e3b150eb3b59cb8a87b73b00e32f18f52fb5842 WatchSource:0}: Error finding container 05bbe83caea16fae2910005b2e3b150eb3b59cb8a87b73b00e32f18f52fb5842: Status 404 returned error can't find the container with id 05bbe83caea16fae2910005b2e3b150eb3b59cb8a87b73b00e32f18f52fb5842
Oct 31 16:31:51.117522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.117484    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:51.117627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.117558    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:51.117627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.117589    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:51.802675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.802634    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" event={"ID":"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3","Type":"ContainerStarted","Data":"9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"}
Oct 31 16:31:51.802897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.802691    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" event={"ID":"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3","Type":"ContainerStarted","Data":"05bbe83caea16fae2910005b2e3b150eb3b59cb8a87b73b00e32f18f52fb5842"}
Oct 31 16:31:51.916022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:51.915960    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podStartSLOduration=1.915944812 podStartE2EDuration="1.915944812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:51.912968765 +0000 UTC m=+1755.612898077" watchObservedRunningTime="2026-10-31 16:31:51.915944812 +0000 UTC m=+1755.615874123"
Oct 31 16:31:52.401692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:52.401660    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:52.408505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:52.408482    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:52.811151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:52.811117    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:52.812227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:52.812157    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:31:53.209698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:53.209613    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"]
Oct 31 16:31:53.209868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:53.209833    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" containerID="cri-o://bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4" gracePeriod=30
Oct 31 16:31:54.006240 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:31:54.006194    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:31:55.419249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.419199    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"]
Oct 31 16:31:55.432436 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.432405    4519 cpu_manager.go:414] "containerMap: removing container" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:55.432663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.432646    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:55.432824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.432811    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:55.432937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.432925    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5af6878-2c2c-48f8-9753-a6da970432d0" containerName="cleanup"
Oct 31 16:31:55.433060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.433049    4519 cpu_manager.go:414] "containerMap: removing container" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:55.433159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.433148    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:55.433325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.433315    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:55.433423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.433413    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:55.433798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.433778    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="821daccc-0a90-4511-9c5e-613589b2681c" containerName="istio-proxy"
Oct 31 16:31:55.433961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.433948    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc30e44-dfd2-417c-83e2-b0b3c5331259" containerName="cleanup"
Oct 31 16:31:55.434098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.434087    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="36aa932f-7edf-4ac1-9a5f-f0f5ba6d7679" containerName="cleanup"
Oct 31 16:31:55.434200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.434190    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a385b3bc-e028-4510-bdee-5265ce04f4a2" containerName="cleanup"
Oct 31 16:31:55.434299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.434290    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0801f85-c125-4608-bb37-a49eb4eab201" containerName="cleanup"
Oct 31 16:31:55.434395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.434385    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="337a5581-c4a2-4cfb-91ac-349830a1d098" containerName="cleanup"
Oct 31 16:31:55.459983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.459952    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.653176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.653012    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-d0755f-openshift-default-dockercfg-bgcqp\""
Oct 31 16:31:55.677945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.670671    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"]
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699376    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699449    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699632    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699690    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699741    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699814    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.699994    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npb2m\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.700114    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.705473 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.700184    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.801877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.801837    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npb2m\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.801916    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.801979    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802031    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802085    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802132    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802186    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802236    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802294    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802511    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802627    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802754 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802739    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802782    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.802875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.802857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.804664 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.804643    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.804755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.804667    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.872651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.858041    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npb2m\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.873401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.873335    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token\") pod \"e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") " pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:55.999884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:55.999841    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:56.338205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.337718    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:31:56.338205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.337784    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:31:56.494657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.494613    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"]
Oct 31 16:31:56.498971 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:31:56.498943    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884f7246_858e_42ac_93d0_97e24b88c8ea.slice/crio-68ca8b94fb9773a62a5f5d72f617c5440d6c00c8a6510feb5caf701c4fbe5d30 WatchSource:0}: Error finding container 68ca8b94fb9773a62a5f5d72f617c5440d6c00c8a6510feb5caf701c4fbe5d30: Status 404 returned error can't find the container with id 68ca8b94fb9773a62a5f5d72f617c5440d6c00c8a6510feb5caf701c4fbe5d30
Oct 31 16:31:56.501697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.501662    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:56.501805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.501740    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:56.501805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.501783    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:56.842245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.842211    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" event={"ID":"884f7246-858e-42ac-93d0-97e24b88c8ea","Type":"ContainerStarted","Data":"688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"}
Oct 31 16:31:56.842432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.842275    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" event={"ID":"884f7246-858e-42ac-93d0-97e24b88c8ea","Type":"ContainerStarted","Data":"68ca8b94fb9773a62a5f5d72f617c5440d6c00c8a6510feb5caf701c4fbe5d30"}
Oct 31 16:31:56.955727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:56.955673    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podStartSLOduration=1.955659361 podStartE2EDuration="1.955659361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:56.954519337 +0000 UTC m=+1760.654448649" watchObservedRunningTime="2026-10-31 16:31:56.955659361 +0000 UTC m=+1760.655588672"
Oct 31 16:31:57.000426 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:57.000397    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:57.005602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:57.005580    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:57.850260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:57.850222    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:57.851164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:57.851141    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:31:58.286856 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:31:58.286820    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b460663_dd16_49f3_bc01_c606f7afe226.slice/crio-bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b460663_dd16_49f3_bc01_c606f7afe226.slice/crio-conmon-bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:31:58.499778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.499752    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:31:58.568951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.568906    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.568976    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569162    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569242    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:31:58.569404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569305    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569341    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569525    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569591    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569668    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l\") pod \"2b460663-dd16-49f3-bc01-c606f7afe226\" (UID: \"2b460663-dd16-49f3-bc01-c606f7afe226\") "
Oct 31 16:31:58.569976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569594    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:31:58.569976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569780    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:31:58.570116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569880    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:31:58.570116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.569951    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:31:58.570916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.570503    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.570916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.570530    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b460663-dd16-49f3-bc01-c606f7afe226-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.570916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.570544    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.570916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.570558    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.570916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.570574    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.571868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.571832    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:31:58.572495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.572440    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "kube-api-access-xjq7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:31:58.573334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.573308    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:31:58.573719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.573696    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token" pod "2b460663-dd16-49f3-bc01-c606f7afe226" (UID: "2b460663-dd16-49f3-bc01-c606f7afe226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:31:58.671687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.671655    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b460663-dd16-49f3-bc01-c606f7afe226-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.671687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.671684    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.671687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.671696    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjq7l\" (UniqueName: \"kubernetes.io/projected/2b460663-dd16-49f3-bc01-c606f7afe226-kube-api-access-xjq7l\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.672013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.671707    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b460663-dd16-49f3-bc01-c606f7afe226-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:31:58.858653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.858574    4519 generic.go:356] "Generic (PLEG): container finished" podID="2b460663-dd16-49f3-bc01-c606f7afe226" containerID="bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4" exitCode=0
Oct 31 16:31:58.858653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.858614    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"
Oct 31 16:31:58.859108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.858650    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" event={"ID":"2b460663-dd16-49f3-bc01-c606f7afe226","Type":"ContainerDied","Data":"bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4"}
Oct 31 16:31:58.859108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.858703    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb" event={"ID":"2b460663-dd16-49f3-bc01-c606f7afe226","Type":"ContainerDied","Data":"561d6bc9574f33180bd313b79fc05e85d6ae9ed19f47bcbf44ee66918af037b6"}
Oct 31 16:31:58.859108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.858729    4519 scope.go:122] "RemoveContainer" containerID="bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4"
Oct 31 16:31:58.870077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.870056    4519 scope.go:122] "RemoveContainer" containerID="bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4"
Oct 31 16:31:58.870367 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:31:58.870348    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4\": container with ID starting with bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4 not found: ID does not exist" containerID="bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4"
Oct 31 16:31:58.870503 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:58.870382    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4"} err="failed to get container status \"bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4\": rpc error: code = NotFound desc = could not find container \"bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4\": container with ID starting with bfa8b2e25dae5b30d062137c4411278b2c2a101c29c73bef44ffb851a3a705b4 not found: ID does not exist"
Oct 31 16:31:59.017921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:59.017880    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"]
Oct 31 16:31:59.022167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:31:59.022141    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-k5grb"]
Oct 31 16:32:00.440156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:00.440124    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" path="/var/lib/kubelet/pods/2b460663-dd16-49f3-bc01-c606f7afe226/volumes"
Oct 31 16:32:01.165521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.159900    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"]
Oct 31 16:32:01.165521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.160940    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy"
Oct 31 16:32:01.165521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.160957    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy"
Oct 31 16:32:01.165521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.161091    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b460663-dd16-49f3-bc01-c606f7afe226" containerName="istio-proxy"
Oct 31 16:32:01.166289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.166264    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.288489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.287385    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3dc1da-openshift-default-dockercfg-spvjh\""
Oct 31 16:32:01.306885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.306849    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"]
Oct 31 16:32:01.314523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314288    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.314523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314365    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.314523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314500    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.314818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314557    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.314818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314692    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.314818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314733    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.314818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314793    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.315044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314837    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.315044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.314908    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tj9\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417304    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417383    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417469    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tj9\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417599    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417691    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417746    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417886    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.417941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.417926    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.418787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.418412    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.419573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.419548    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.419752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.419609    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.420423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.419840    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.420861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.420835    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.421976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.421951    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.423428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.423389    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.482754 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.482722    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.483278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.482734    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tj9\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9\") pod \"e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") " pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:01.497704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:01.497665    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:02.071747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:02.046661    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"]
Oct 31 16:32:02.223669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:02.220030    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:02.223669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:02.220155    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:02.223669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:02.223440    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:02.897387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:02.897346    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" event={"ID":"9393cbea-c7d6-47d7-9a20-a52a318ef5e6","Type":"ContainerStarted","Data":"9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"}
Oct 31 16:32:02.898131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:02.898091    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" event={"ID":"9393cbea-c7d6-47d7-9a20-a52a318ef5e6","Type":"ContainerStarted","Data":"aec57ce937c9df4190fc203d32da08bd674921038f2ab7a47cda09c3d10deff1"}
Oct 31 16:32:03.014019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:03.013521    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podStartSLOduration=2.013499945 podStartE2EDuration="2.013499945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:03.008929512 +0000 UTC m=+1766.708858825" watchObservedRunningTime="2026-10-31 16:32:03.013499945 +0000 UTC m=+1766.713429259"
Oct 31 16:32:03.498195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:03.498158    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:03.503826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:03.503800    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:03.904587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:03.904498    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:03.905623 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:03.905599    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:32:13.103182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.103145    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"]
Oct 31 16:32:13.117611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.117575    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.150699 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.150664    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"]
Oct 31 16:32:13.168582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.168552    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdc828-openshift-default-dockercfg-gq77f\""
Oct 31 16:32:13.225846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.225813    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.225846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.225848    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.225947    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv76w\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226110 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.226089    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.226121    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.226168    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226298 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.226277    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.226333    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.226469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.226438    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.327932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.327898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328017    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328037    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328103    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv76w\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328183    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328203    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328332    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328575    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328603    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328630    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328690    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328817    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.328889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.328861    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.329337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.329314    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.330437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.330413    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.330541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.330509    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.382312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.382246    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv76w\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.382312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.382259    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-grp6f\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.438431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.438398    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:13.892484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.886898    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"]
Oct 31 16:32:13.902749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.901858    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:13.902749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.901958    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:13.902749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:13.902003    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:14.084567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:14.084531    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" event={"ID":"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a","Type":"ContainerStarted","Data":"fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"}
Oct 31 16:32:14.084770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:14.084597    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" event={"ID":"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a","Type":"ContainerStarted","Data":"6d1938c6c6bb59fcf6d0b4325c9b338681400cba3674660fd72ab2d2d76839d8"}
Oct 31 16:32:14.197906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:14.197789    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podStartSLOduration=1.197773749 podStartE2EDuration="1.197773749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:14.196030735 +0000 UTC m=+1777.895960062" watchObservedRunningTime="2026-10-31 16:32:14.197773749 +0000 UTC m=+1777.897703060"
Oct 31 16:32:14.462984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:14.461717    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:15.444947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:15.444917    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:16.103048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:16.103016    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:16.104058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:16.104032    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:32:19.180820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.180775    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"]
Oct 31 16:32:19.185872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.185849    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.231711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.231635    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"]
Oct 31 16:32:19.238122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.238094    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.277575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.277543    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"]
Oct 31 16:32:19.358020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.357902    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.358342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358319    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.358575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358550    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.358703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358626    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vvc\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.358767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358730    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"]
Oct 31 16:32:19.358819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358808    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.358887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358869    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.358981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.358960    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.359093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359068    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.359160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359108    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.359215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359197    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.359263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359245    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.359322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359272    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28q8j\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.359322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359299    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.359429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359321    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.359429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359352    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.359651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359628    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.359764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359711    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.359849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.359783    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.461790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.461765    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.461956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.461816    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.461956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.461875    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.461956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.461895    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.461975    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462060    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462198    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462212    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462254    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462330    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vvc\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462354    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462450    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462538    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462586    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462654    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462663    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462681    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462727    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462782    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.462863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462814    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28q8j\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.462863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462849    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.463055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462876    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.463055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462907    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.463055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.462983    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.463254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.463230    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.463325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.463297    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.463510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.463488    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.463593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.463573    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.464821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.464785    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.464944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.464811    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.465229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.465200    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.465614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.465593    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.516246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.516218    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vvc\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.516551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.516529    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.516666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.516589    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28q8j\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.516791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.516774    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.557296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.557271    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:19.803665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.803574    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:19.989963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.989932    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"]
Oct 31 16:32:19.993114 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:32:19.993088    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4531531c_a5f7_45da_ae82_e00f6fc372cd.slice/crio-eac94c3a087a719a7f28074157731d1f3c1a7e9e7a675ae44aa183d1b908a611 WatchSource:0}: Error finding container eac94c3a087a719a7f28074157731d1f3c1a7e9e7a675ae44aa183d1b908a611: Status 404 returned error can't find the container with id eac94c3a087a719a7f28074157731d1f3c1a7e9e7a675ae44aa183d1b908a611
Oct 31 16:32:19.997256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.997214    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:19.997363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.997301    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:19.997363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:19.997345    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.138098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.138056    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" event={"ID":"4531531c-a5f7-45da-ae82-e00f6fc372cd","Type":"ContainerStarted","Data":"2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017"}
Oct 31 16:32:20.138281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.138122    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" event={"ID":"4531531c-a5f7-45da-ae82-e00f6fc372cd","Type":"ContainerStarted","Data":"eac94c3a087a719a7f28074157731d1f3c1a7e9e7a675ae44aa183d1b908a611"}
Oct 31 16:32:20.189537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.189489    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"]
Oct 31 16:32:20.194630 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:32:20.194598    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861b064c_4ced_4226_aaee_915a2c9c4019.slice/crio-1c2d2677f0198b041e1c0bd4dd7ec0ade2cefc23ee8fab0d884170f4bc5ebd82 WatchSource:0}: Error finding container 1c2d2677f0198b041e1c0bd4dd7ec0ade2cefc23ee8fab0d884170f4bc5ebd82: Status 404 returned error can't find the container with id 1c2d2677f0198b041e1c0bd4dd7ec0ade2cefc23ee8fab0d884170f4bc5ebd82
Oct 31 16:32:20.197353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.197321    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.197471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.197382    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.197471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.197416    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.255357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.255309    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podStartSLOduration=1.255294833 podStartE2EDuration="1.255294833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:20.251770491 +0000 UTC m=+1783.951699824" watchObservedRunningTime="2026-10-31 16:32:20.255294833 +0000 UTC m=+1783.955224143"
Oct 31 16:32:20.557924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:20.557883    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:21.147076 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.147040    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" event={"ID":"861b064c-4ced-4226-aaee-915a2c9c4019","Type":"ContainerStarted","Data":"d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4"}
Oct 31 16:32:21.147254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.147105    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" event={"ID":"861b064c-4ced-4226-aaee-915a2c9c4019","Type":"ContainerStarted","Data":"1c2d2677f0198b041e1c0bd4dd7ec0ade2cefc23ee8fab0d884170f4bc5ebd82"}
Oct 31 16:32:21.351308 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.351259    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podStartSLOduration=2.351244494 podStartE2EDuration="2.351244494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:21.30396015 +0000 UTC m=+1785.003889459" watchObservedRunningTime="2026-10-31 16:32:21.351244494 +0000 UTC m=+1785.051173806"
Oct 31 16:32:21.559319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.559287    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 16:32:21.559549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.559345    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 16:32:21.808478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.807623    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:21.818275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.817810    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 16:32:21.818275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:21.817889    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 16:32:22.569489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:22.568332    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 16:32:22.569489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:22.568395    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 16:32:22.875377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:22.851832    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 16:32:22.889501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:22.855532    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 16:32:23.558145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:23.558112    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 16:32:23.558333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:23.558164    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 16:32:23.805751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:23.805713    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 16:32:23.806324 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:23.806290    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 16:32:24.058318 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:32:24.058284    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:32:24.558255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:24.558221    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 16:32:24.558605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:24.558571    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 16:32:24.804866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:24.804827    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 16:32:24.805153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:24.805121    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 16:32:25.557811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:25.557776    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 16:32:25.558249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:25.557831    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 16:32:25.807984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:25.807885    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 16:32:25.808276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:25.808238    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 16:32:26.558514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:26.558484    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 16:32:26.558967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:26.558937    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 16:32:26.803989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:26.803947    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 16:32:26.804205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:26.804009    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 16:32:27.562857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:27.562826    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:27.563310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:27.563291    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:27.564056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:27.564038    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:32:27.809601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:27.809566    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:27.809863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:27.809825    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:27.810505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:27.810487    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:32:37.044966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:37.044935    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"]
Oct 31 16:32:37.092113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:37.092084    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-102e1ef5-lwx7k"]
Oct 31 16:32:38.333690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:38.333658    4519 scope.go:122] "RemoveContainer" containerID="88ed992513b9116a54a529f234406c197d921a89fe28c73c14e6afed737d5c71"
Oct 31 16:32:38.344746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:38.344703    4519 scope.go:122] "RemoveContainer" containerID="b449aacd8e7c773d1cef8aed072b6078c177915c7984f172bf3e54b1f38cc959"
Oct 31 16:32:38.441415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:38.441385    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4660b8d3-720d-4c97-8ff9-e1325a82ffb8" path="/var/lib/kubelet/pods/4660b8d3-720d-4c97-8ff9-e1325a82ffb8/volumes"
Oct 31 16:32:47.465998 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.465967    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"]
Oct 31 16:32:47.472206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.472186    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.538550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.524253    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a2ae37-openshift-default-dockercfg-kxvqq\""
Oct 31 16:32:47.620612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.620537    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"]
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625495    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625613    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625759    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625807    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625854    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625878    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.625999    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.643519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.626142    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bb98\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.786872    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.786955    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787115    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bb98\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787176    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787446    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787569    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.787592    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.788139    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.788380    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.789403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.788967    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.882777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.798521    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.882777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.799171    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.902514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.898793    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.912152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.912116    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.913999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.913974    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:47.914604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:47.914554    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bb98\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98\") pod \"e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") " pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:48.088119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:48.088040    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:48.570749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:48.570716    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"]
Oct 31 16:32:48.574691 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:32:48.574661    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3650035e_9a85_4eee_8dd4_a2bab7815442.slice/crio-77af27a8505028ab20c14e55a89b3216fedbea8c3f41055e71deec85bd0b8ee2 WatchSource:0}: Error finding container 77af27a8505028ab20c14e55a89b3216fedbea8c3f41055e71deec85bd0b8ee2: Status 404 returned error can't find the container with id 77af27a8505028ab20c14e55a89b3216fedbea8c3f41055e71deec85bd0b8ee2
Oct 31 16:32:48.577520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:48.577484    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:48.577630 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:48.577559    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:48.577630 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:48.577590    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:49.563100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:49.563061    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" event={"ID":"3650035e-9a85-4eee-8dd4-a2bab7815442","Type":"ContainerStarted","Data":"1adf9c6c1dfd7eb4a25219c6f60dc05c95d4136faf953e407d40645884253e10"}
Oct 31 16:32:49.563277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:49.563177    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" event={"ID":"3650035e-9a85-4eee-8dd4-a2bab7815442","Type":"ContainerStarted","Data":"77af27a8505028ab20c14e55a89b3216fedbea8c3f41055e71deec85bd0b8ee2"}
Oct 31 16:32:49.725042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:49.724992    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podStartSLOduration=2.724977761 podStartE2EDuration="2.724977761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:49.720553313 +0000 UTC m=+1813.420482637" watchObservedRunningTime="2026-10-31 16:32:49.724977761 +0000 UTC m=+1813.424907072"
Oct 31 16:32:50.088305 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:50.088271    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:50.094271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:50.094246    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:50.570632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:50.570594    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:50.571604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:50.571580    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:32:52.026256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:32:52.026223    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:32:54.005625 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:32:54.005578    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:33:24.009779 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:33:24.009681    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:33:37.272222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.272185    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"]
Oct 31 16:33:37.276758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.276736    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.327588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.327561    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"]
Oct 31 16:33:37.369703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.369675    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:33:37.369872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.369749    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:33:37.369872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.369771    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:33:37.373211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.373194    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:33:37.401427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.401400    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.401691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.401670    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jckw\" (UniqueName: \"kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.401752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.401716    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.503247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.503216    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.503412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.503387    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jckw\" (UniqueName: \"kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.503489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.503419    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.503903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.503883    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.505689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.505672    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.557040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.556976    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jckw\" (UniqueName: \"kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.591469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.591422    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:37.967779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.967745    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"]
Oct 31 16:33:37.973620 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:33:37.973592    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6635f83_c118_452d_9e8a_ddf372396c2a.slice/crio-b83e7e45bf0afb0ef3547a423554cd86701dd2efdfb2579d6469b2f302eb655e WatchSource:0}: Error finding container b83e7e45bf0afb0ef3547a423554cd86701dd2efdfb2579d6469b2f302eb655e: Status 404 returned error can't find the container with id b83e7e45bf0afb0ef3547a423554cd86701dd2efdfb2579d6469b2f302eb655e
Oct 31 16:33:37.983053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:37.983024    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck" event={"ID":"a6635f83-c118-452d-9e8a-ddf372396c2a","Type":"ContainerStarted","Data":"b83e7e45bf0afb0ef3547a423554cd86701dd2efdfb2579d6469b2f302eb655e"}
Oct 31 16:33:38.381144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:38.381118    4519 scope.go:122] "RemoveContainer" containerID="9ef63a9bbb1823188647516416fcab1a09216f7ae99f9efe3b8dd4a39b34309e"
Oct 31 16:33:38.970330 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:33:38.970299    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6635f83_c118_452d_9e8a_ddf372396c2a.slice/crio-29d1cc44b94f8a27832ae6ba253cff0587d46df67695d5445ce4b91870092464.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:33:38.992612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:38.992577    4519 generic.go:356] "Generic (PLEG): container finished" podID="a6635f83-c118-452d-9e8a-ddf372396c2a" containerID="29d1cc44b94f8a27832ae6ba253cff0587d46df67695d5445ce4b91870092464" exitCode=0
Oct 31 16:33:38.992774 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:38.992627    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck" event={"ID":"a6635f83-c118-452d-9e8a-ddf372396c2a","Type":"ContainerDied","Data":"29d1cc44b94f8a27832ae6ba253cff0587d46df67695d5445ce4b91870092464"}
Oct 31 16:33:39.358430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.358395    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"]
Oct 31 16:33:39.362992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.362971    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.415911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.415877    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"]
Oct 31 16:33:39.538632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.538575    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.538825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.538709    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.538825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.538753    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zh6l\" (UniqueName: \"kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.640289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.640190    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.640289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.640282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.640532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.640326    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zh6l\" (UniqueName: \"kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.641061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.641033    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.642730 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.642705    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.694729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.694699    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zh6l\" (UniqueName: \"kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l\") pod \"maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:39.976674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:39.976644    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:40.198246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.198220    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:40.353519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.353396    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw\" (UniqueName: \"kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw\") pod \"a6635f83-c118-452d-9e8a-ddf372396c2a\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") "
Oct 31 16:33:40.353519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.353499    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca\") pod \"a6635f83-c118-452d-9e8a-ddf372396c2a\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") "
Oct 31 16:33:40.353812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.353664    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca\") pod \"a6635f83-c118-452d-9e8a-ddf372396c2a\" (UID: \"a6635f83-c118-452d-9e8a-ddf372396c2a\") "
Oct 31 16:33:40.354197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.354167    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca" pod "a6635f83-c118-452d-9e8a-ddf372396c2a" (UID: "a6635f83-c118-452d-9e8a-ddf372396c2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:33:40.354544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.354525    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:40.356187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.356166    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca" pod "a6635f83-c118-452d-9e8a-ddf372396c2a" (UID: "a6635f83-c118-452d-9e8a-ddf372396c2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:33:40.356293 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.356263    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw" pod "a6635f83-c118-452d-9e8a-ddf372396c2a" (UID: "a6635f83-c118-452d-9e8a-ddf372396c2a"). InnerVolumeSpecName "kube-api-access-4jckw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:33:40.359229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.359130    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"]
Oct 31 16:33:40.361995 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:33:40.361970    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0948af8f_66a8_48e2_ad44_1e1cf333cd0c.slice/crio-38ca28804016b2976abdb11d03eff39b2e82b8208e157c4377abade98b02df94 WatchSource:0}: Error finding container 38ca28804016b2976abdb11d03eff39b2e82b8208e157c4377abade98b02df94: Status 404 returned error can't find the container with id 38ca28804016b2976abdb11d03eff39b2e82b8208e157c4377abade98b02df94
Oct 31 16:33:40.456610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.456585    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jckw\" (UniqueName: \"kubernetes.io/projected/a6635f83-c118-452d-9e8a-ddf372396c2a-kube-api-access-4jckw\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:40.456610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:40.456610    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6635f83-c118-452d-9e8a-ddf372396c2a-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:41.010222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:41.010194    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"
Oct 31 16:33:41.010507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:41.010216    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck" event={"ID":"a6635f83-c118-452d-9e8a-ddf372396c2a","Type":"ContainerDied","Data":"b83e7e45bf0afb0ef3547a423554cd86701dd2efdfb2579d6469b2f302eb655e"}
Oct 31 16:33:41.010655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:41.010550    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b83e7e45bf0afb0ef3547a423554cd86701dd2efdfb2579d6469b2f302eb655e"
Oct 31 16:33:41.012322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:41.012298    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv" event={"ID":"0948af8f-66a8-48e2-ad44-1e1cf333cd0c","Type":"ContainerStarted","Data":"1bd0a8b83d88f2eb1afe89c9d791ba9f2874e90d90eeaa76a05e5ce4d9d70f11"}
Oct 31 16:33:41.012434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:41.012345    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv" event={"ID":"0948af8f-66a8-48e2-ad44-1e1cf333cd0c","Type":"ContainerStarted","Data":"38ca28804016b2976abdb11d03eff39b2e82b8208e157c4377abade98b02df94"}
Oct 31 16:33:41.122177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:41.122117    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv" podStartSLOduration=2.122102641 podStartE2EDuration="2.122102641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:41.118789073 +0000 UTC m=+1864.818718384" watchObservedRunningTime="2026-10-31 16:33:41.122102641 +0000 UTC m=+1864.822031969"
Oct 31 16:33:42.022703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:42.022666    4519 generic.go:356] "Generic (PLEG): container finished" podID="0948af8f-66a8-48e2-ad44-1e1cf333cd0c" containerID="1bd0a8b83d88f2eb1afe89c9d791ba9f2874e90d90eeaa76a05e5ce4d9d70f11" exitCode=0
Oct 31 16:33:42.023104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:42.022739    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv" event={"ID":"0948af8f-66a8-48e2-ad44-1e1cf333cd0c","Type":"ContainerDied","Data":"1bd0a8b83d88f2eb1afe89c9d791ba9f2874e90d90eeaa76a05e5ce4d9d70f11"}
Oct 31 16:33:43.238362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.238338    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:43.305117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.305041    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca\") pod \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") "
Oct 31 16:33:43.305117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.305081    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca\") pod \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") "
Oct 31 16:33:43.305369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.305128    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l\" (UniqueName: \"kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l\") pod \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\" (UID: \"0948af8f-66a8-48e2-ad44-1e1cf333cd0c\") "
Oct 31 16:33:43.305569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.305544    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca" pod "0948af8f-66a8-48e2-ad44-1e1cf333cd0c" (UID: "0948af8f-66a8-48e2-ad44-1e1cf333cd0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:33:43.306141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.306112    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:43.307226 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.307204    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca" pod "0948af8f-66a8-48e2-ad44-1e1cf333cd0c" (UID: "0948af8f-66a8-48e2-ad44-1e1cf333cd0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:33:43.307327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.307289    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l" pod "0948af8f-66a8-48e2-ad44-1e1cf333cd0c" (UID: "0948af8f-66a8-48e2-ad44-1e1cf333cd0c"). InnerVolumeSpecName "kube-api-access-2zh6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:33:43.408247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.408208    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:43.408247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:43.408241    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zh6l\" (UniqueName: \"kubernetes.io/projected/0948af8f-66a8-48e2-ad44-1e1cf333cd0c-kube-api-access-2zh6l\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:44.039881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:44.039847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv" event={"ID":"0948af8f-66a8-48e2-ad44-1e1cf333cd0c","Type":"ContainerDied","Data":"38ca28804016b2976abdb11d03eff39b2e82b8208e157c4377abade98b02df94"}
Oct 31 16:33:44.039881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:44.039878    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"
Oct 31 16:33:44.040100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:44.039894    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ca28804016b2976abdb11d03eff39b2e82b8208e157c4377abade98b02df94"
Oct 31 16:33:44.714528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:44.714490    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:33:53.854712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:53.854665    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:33:53.855229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:53.854740    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:33:54.019092 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:33:54.019048    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:33:56.125575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.125541    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"]
Oct 31 16:33:56.126040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.125738    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy" containerID="cri-o://2eac6679292e49b8e3619d5d5a4513ac0bb3d037cf81a2597304f00f4abff0b8" gracePeriod=30
Oct 31 16:33:56.210418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.210391    4519 generic.go:356] "Generic (PLEG): container finished" podID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerID="9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520" exitCode=0
Oct 31 16:33:56.210587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.210505    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" event={"ID":"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3","Type":"ContainerDied","Data":"9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"}
Oct 31 16:33:56.211111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.211094    4519 scope.go:122] "RemoveContainer" containerID="9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"
Oct 31 16:33:56.213874 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.213819    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:33:56.214065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.214036    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:33:56.214175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:56.214150    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:33:57.220257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:57.220219    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" event={"ID":"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3","Type":"ContainerStarted","Data":"9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631"}
Oct 31 16:33:57.402493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:57.402451    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:33:57.402663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:57.402610    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:33:57.402707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:57.402669    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 16:33:58.402158 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:58.402122    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:33:58.402672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:58.402191    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 16:33:59.402008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:59.401978    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:33:59.402203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:33:59.402033    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 16:34:00.401932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:00.401897    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:34:00.402130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:00.401962    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 16:34:01.251603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.251575    4519 generic.go:356] "Generic (PLEG): container finished" podID="8019d988-49e9-448b-a12a-ad8b58514934" containerID="2eac6679292e49b8e3619d5d5a4513ac0bb3d037cf81a2597304f00f4abff0b8" exitCode=0
Oct 31 16:34:01.251948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.251652    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" event={"ID":"8019d988-49e9-448b-a12a-ad8b58514934","Type":"ContainerDied","Data":"2eac6679292e49b8e3619d5d5a4513ac0bb3d037cf81a2597304f00f4abff0b8"}
Oct 31 16:34:01.402553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.402194    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:34:01.402553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.402253    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 16:34:01.456897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.456873    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:34:01.555954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.555923    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.555981    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556091    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556150    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556225    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556373    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556427    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556446    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:01.556615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556512    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556639    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token\") pod \"8019d988-49e9-448b-a12a-ad8b58514934\" (UID: \"8019d988-49e9-448b-a12a-ad8b58514934\") "
Oct 31 16:34:01.556673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556659    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:01.556797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556681    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:01.556797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556722    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:01.556973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.556819    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:01.557719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.557620    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.557719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.557646    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8019d988-49e9-448b-a12a-ad8b58514934-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.557719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.557662    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.557719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.557678    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.557719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.557695    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.558400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.558379    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "kube-api-access-thdxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:01.558732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.558678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:01.558732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.558710    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:34:01.559070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.559047    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token" pod "8019d988-49e9-448b-a12a-ad8b58514934" (UID: "8019d988-49e9-448b-a12a-ad8b58514934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:01.722221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.722113    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8019d988-49e9-448b-a12a-ad8b58514934-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.722221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.722165    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.722221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.722180    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8019d988-49e9-448b-a12a-ad8b58514934-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:01.722729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:01.722194    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thdxk\" (UniqueName: \"kubernetes.io/projected/8019d988-49e9-448b-a12a-ad8b58514934-kube-api-access-thdxk\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:02.260653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.260622    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"
Oct 31 16:34:02.261098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.260647    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j" event={"ID":"8019d988-49e9-448b-a12a-ad8b58514934","Type":"ContainerDied","Data":"c09a0c4994e83c6ec393be560e1adf3e771100abab2b6eff51110a750e97db8e"}
Oct 31 16:34:02.261098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.260713    4519 scope.go:122] "RemoveContainer" containerID="2eac6679292e49b8e3619d5d5a4513ac0bb3d037cf81a2597304f00f4abff0b8"
Oct 31 16:34:02.401670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.401634    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:34:02.401840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.401693    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 16:34:02.419868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.419830    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"]
Oct 31 16:34:02.491763 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:02.491723    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zgr9j"]
Oct 31 16:34:03.406658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:03.406629    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:34:03.407153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:03.407131    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:34:03.407937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:03.407916    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:34:04.440553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:04.440515    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8019d988-49e9-448b-a12a-ad8b58514934" path="/var/lib/kubelet/pods/8019d988-49e9-448b-a12a-ad8b58514934/volumes"
Oct 31 16:34:09.049114 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.049081    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"]
Oct 31 16:34:09.049942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.049922    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy"
Oct 31 16:34:09.050005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.049943    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy"
Oct 31 16:34:09.050005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.049952    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0948af8f-66a8-48e2-ad44-1e1cf333cd0c" containerName="revoke-keys"
Oct 31 16:34:09.050005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.049957    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0948af8f-66a8-48e2-ad44-1e1cf333cd0c" containerName="revoke-keys"
Oct 31 16:34:09.050005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.049999    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a6635f83-c118-452d-9e8a-ddf372396c2a" containerName="revoke-keys"
Oct 31 16:34:09.050005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.050004    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6635f83-c118-452d-9e8a-ddf372396c2a" containerName="revoke-keys"
Oct 31 16:34:09.050178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.050130    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6635f83-c118-452d-9e8a-ddf372396c2a" containerName="revoke-keys"
Oct 31 16:34:09.050178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.050143    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8019d988-49e9-448b-a12a-ad8b58514934" containerName="istio-proxy"
Oct 31 16:34:09.050178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.050149    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0948af8f-66a8-48e2-ad44-1e1cf333cd0c" containerName="revoke-keys"
Oct 31 16:34:09.055120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.055102    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.070510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.070486    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.070616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.070591    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.070780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.070763    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8h62\" (UniqueName: \"kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.105044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.105018    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:34:09.105478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.105437    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"]
Oct 31 16:34:09.147100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.147075    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:34:09.147516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.147445    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:34:09.147630 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.147543    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:34:09.171739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.171709    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8h62\" (UniqueName: \"kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.171913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.171811    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.171913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.171889    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.172526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.172506    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.174267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.174247    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.225864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.225832    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8h62\" (UniqueName: \"kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62\") pod \"maas-api-revoke-sub-8e5c1e77-82nwx\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.369759 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.369684    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:09.748479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:09.748434    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"]
Oct 31 16:34:09.753488 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:34:09.753425    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290b5587_0512_44c5_8e6a_00634ee620bb.slice/crio-cc9a1185070a42f09d3cbaabc04bd7dcb45b61e5b5c755a441a486829fd90a5d WatchSource:0}: Error finding container cc9a1185070a42f09d3cbaabc04bd7dcb45b61e5b5c755a441a486829fd90a5d: Status 404 returned error can't find the container with id cc9a1185070a42f09d3cbaabc04bd7dcb45b61e5b5c755a441a486829fd90a5d
Oct 31 16:34:10.321222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:10.321186    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx" event={"ID":"290b5587-0512-44c5-8e6a-00634ee620bb","Type":"ContainerStarted","Data":"6511a0bc64755220f2aeaab5e3bfe25b173e267173b4fd5b0f9913b550d1cf82"}
Oct 31 16:34:10.321649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:10.321237    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx" event={"ID":"290b5587-0512-44c5-8e6a-00634ee620bb","Type":"ContainerStarted","Data":"cc9a1185070a42f09d3cbaabc04bd7dcb45b61e5b5c755a441a486829fd90a5d"}
Oct 31 16:34:10.428768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:10.428654    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx" podStartSLOduration=1.428637867 podStartE2EDuration="1.428637867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:10.42813331 +0000 UTC m=+1894.128062623" watchObservedRunningTime="2026-10-31 16:34:10.428637867 +0000 UTC m=+1894.128567202"
Oct 31 16:34:11.330085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:11.330050    4519 generic.go:356] "Generic (PLEG): container finished" podID="290b5587-0512-44c5-8e6a-00634ee620bb" containerID="6511a0bc64755220f2aeaab5e3bfe25b173e267173b4fd5b0f9913b550d1cf82" exitCode=0
Oct 31 16:34:11.330581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:11.330139    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx" event={"ID":"290b5587-0512-44c5-8e6a-00634ee620bb","Type":"ContainerDied","Data":"6511a0bc64755220f2aeaab5e3bfe25b173e267173b4fd5b0f9913b550d1cf82"}
Oct 31 16:34:12.516839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.516811    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:12.625115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.625034    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca\") pod \"290b5587-0512-44c5-8e6a-00634ee620bb\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") "
Oct 31 16:34:12.625273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.625129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62\" (UniqueName: \"kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62\") pod \"290b5587-0512-44c5-8e6a-00634ee620bb\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") "
Oct 31 16:34:12.625430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.625409    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca\") pod \"290b5587-0512-44c5-8e6a-00634ee620bb\" (UID: \"290b5587-0512-44c5-8e6a-00634ee620bb\") "
Oct 31 16:34:12.626594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.626564    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca" pod "290b5587-0512-44c5-8e6a-00634ee620bb" (UID: "290b5587-0512-44c5-8e6a-00634ee620bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:12.626879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.626859    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:12.627915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.627824    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca" pod "290b5587-0512-44c5-8e6a-00634ee620bb" (UID: "290b5587-0512-44c5-8e6a-00634ee620bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:34:12.627915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.627864    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62" pod "290b5587-0512-44c5-8e6a-00634ee620bb" (UID: "290b5587-0512-44c5-8e6a-00634ee620bb"). InnerVolumeSpecName "kube-api-access-b8h62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:12.728289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.728255    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290b5587-0512-44c5-8e6a-00634ee620bb-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:12.728289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:12.728286    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8h62\" (UniqueName: \"kubernetes.io/projected/290b5587-0512-44c5-8e6a-00634ee620bb-kube-api-access-b8h62\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:13.346435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:13.346400    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx" event={"ID":"290b5587-0512-44c5-8e6a-00634ee620bb","Type":"ContainerDied","Data":"cc9a1185070a42f09d3cbaabc04bd7dcb45b61e5b5c755a441a486829fd90a5d"}
Oct 31 16:34:13.346435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:13.346435    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"
Oct 31 16:34:13.346673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:13.346466    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc9a1185070a42f09d3cbaabc04bd7dcb45b61e5b5c755a441a486829fd90a5d"
Oct 31 16:34:24.006041 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:34:24.005993    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:34:30.228198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:30.228159    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:34:38.410528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:38.410496    4519 scope.go:122] "RemoveContainer" containerID="44a03de651a0c9411ac95bcc42717a86e1066be8673e5658f6ae858c49232f6b"
Oct 31 16:34:38.424980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:38.424957    4519 scope.go:122] "RemoveContainer" containerID="1fe00d9161736b119c815ee7ab9983c7827fbfa977d9401c12087633afbe0fe4"
Oct 31 16:34:41.470490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:41.470424    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"]
Oct 31 16:34:41.470901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:41.470724    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" containerID="cri-o://7587a643026f6ca5459d9efed901916005d9d41444cea0c583cd10212c8fb2c9" gracePeriod=30
Oct 31 16:34:41.731390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:41.731309    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"]
Oct 31 16:34:41.733807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:41.733751    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" containerID="cri-o://15bc41744296e6640d8ba30a2c375696a8d1ef9c3c4423278c864944d0eb99bd" gracePeriod=30
Oct 31 16:34:43.264016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:43.263983    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:34:43.297438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:43.264047    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:34:43.891684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:43.891589    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:34:43.891684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:43.891664    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:34:44.719148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.719117    4519 generic.go:356] "Generic (PLEG): container finished" podID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerID="688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551" exitCode=0
Oct 31 16:34:44.719642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.719190    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" event={"ID":"884f7246-858e-42ac-93d0-97e24b88c8ea","Type":"ContainerDied","Data":"688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"}
Oct 31 16:34:44.719749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.719729    4519 scope.go:122] "RemoveContainer" containerID="688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"
Oct 31 16:34:44.723672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.723626    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.723829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.723804    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.723947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.723913    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.737072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.737036    4519 generic.go:356] "Generic (PLEG): container finished" podID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerID="9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f" exitCode=0
Oct 31 16:34:44.737390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.737236    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" event={"ID":"9393cbea-c7d6-47d7-9a20-a52a318ef5e6","Type":"ContainerDied","Data":"9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"}
Oct 31 16:34:44.737845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.737821    4519 scope.go:122] "RemoveContainer" containerID="9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"
Oct 31 16:34:44.739268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.739245    4519 generic.go:356] "Generic (PLEG): container finished" podID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerID="fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829" exitCode=0
Oct 31 16:34:44.739386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.739323    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" event={"ID":"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a","Type":"ContainerDied","Data":"fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"}
Oct 31 16:34:44.739735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.739717    4519 scope.go:122] "RemoveContainer" containerID="fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"
Oct 31 16:34:44.745055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.745025    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.745165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.745045    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.745165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.745092    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.745165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.745109    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.745165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.745124    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:44.745165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:44.745141    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:45.752714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:45.752676    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" event={"ID":"884f7246-858e-42ac-93d0-97e24b88c8ea","Type":"ContainerStarted","Data":"aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df"}
Oct 31 16:34:45.756420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:45.756370    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" event={"ID":"9393cbea-c7d6-47d7-9a20-a52a318ef5e6","Type":"ContainerStarted","Data":"f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4"}
Oct 31 16:34:45.759384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:45.759352    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" event={"ID":"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a","Type":"ContainerStarted","Data":"a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606"}
Oct 31 16:34:46.004537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.003447    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:34:46.017479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.012530    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:34:46.017479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.012611    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 16:34:46.123480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.119534    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:34:46.456785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.456370    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 16:34:46.456785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.456429    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 16:34:46.464138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.464113    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:34:46.503008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.501813    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:34:46.503008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.501947    4519 patch_prober.go:29] interesting pod/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 16:34:46.503008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.501990    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 16:34:46.802585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.802551    4519 generic.go:356] "Generic (PLEG): container finished" podID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerID="7587a643026f6ca5459d9efed901916005d9d41444cea0c583cd10212c8fb2c9" exitCode=0
Oct 31 16:34:46.804390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:46.804360    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" event={"ID":"9846e224-aeeb-4612-bb4b-e66f8366c1dc","Type":"ContainerDied","Data":"7587a643026f6ca5459d9efed901916005d9d41444cea0c583cd10212c8fb2c9"}
Oct 31 16:34:47.009481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.008852    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:34:47.009481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.008911    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 16:34:47.461853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.449519    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 16:34:47.461853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.449575    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 16:34:47.461853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.451921    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mb94r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 31 16:34:47.461853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.451962    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 31 16:34:47.555712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.555674    4519 patch_prober.go:29] interesting pod/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 16:34:47.555904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:47.555741    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 16:34:48.036488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.029767    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:34:48.036488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.029827    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 16:34:48.098477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.097672    4519 generic.go:356] "Generic (PLEG): container finished" podID="0a759655-221d-4180-af64-e8bf585bcbdb" containerID="15bc41744296e6640d8ba30a2c375696a8d1ef9c3c4423278c864944d0eb99bd" exitCode=0
Oct 31 16:34:48.123479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.099010    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" event={"ID":"0a759655-221d-4180-af64-e8bf585bcbdb","Type":"ContainerDied","Data":"15bc41744296e6640d8ba30a2c375696a8d1ef9c3c4423278c864944d0eb99bd"}
Oct 31 16:34:48.320531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.319637    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:34:48.346481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.334641    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480155    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480250    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480299    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480328    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480355    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480396    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480560    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480603    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480713    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480778    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480821    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.480951    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.481019    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.481067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.481162    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.481196    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo\") pod \"0a759655-221d-4180-af64-e8bf585bcbdb\" (UID: \"0a759655-221d-4180-af64-e8bf585bcbdb\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.481233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.496491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.481317    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs\") pod \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\" (UID: \"9846e224-aeeb-4612-bb4b-e66f8366c1dc\") "
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.517086    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.517497    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.519686    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.519892    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.520592    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.520647    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.520689    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.539923    4519 patch_prober.go:29] interesting pod/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.539991    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.540250    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.540399    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.541506    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.552840    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.552952    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:34:48.562550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.553716    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.564694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.564646    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.593528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.587695    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:48.593528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.587813    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:48.672200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.650698    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:48.672580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.672508    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:48.683990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.683954    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:48.684227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.684206    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:48.684349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.684334    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:48.684470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.684442    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a759655-221d-4180-af64-e8bf585bcbdb-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:48.923834    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.036928    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.086755    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.097516    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt" pod "0a759655-221d-4180-af64-e8bf585bcbdb" (UID: "0a759655-221d-4180-af64-e8bf585bcbdb"). InnerVolumeSpecName "kube-api-access-scvwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.148717    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.149720    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.151006    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm" pod "9846e224-aeeb-4612-bb4b-e66f8366c1dc" (UID: "9846e224-aeeb-4612-bb4b-e66f8366c1dc"). InnerVolumeSpecName "kube-api-access-bmcpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.148581    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153713    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153733    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153748    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153785    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153801    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153823    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.153922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.153838    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a759655-221d-4180-af64-e8bf585bcbdb-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.253628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.253405    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"
Oct 31 16:34:49.253628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.253509    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8" event={"ID":"0a759655-221d-4180-af64-e8bf585bcbdb","Type":"ContainerDied","Data":"bc00d393d7084a95f032c946e6fbde7cbde6a0608092364df1c97a0893a0b3eb"}
Oct 31 16:34:49.253628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.253560    4519 scope.go:122] "RemoveContainer" containerID="15bc41744296e6640d8ba30a2c375696a8d1ef9c3c4423278c864944d0eb99bd"
Oct 31 16:34:49.255793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.255766    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a759655-221d-4180-af64-e8bf585bcbdb-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.255921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.255803    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvwt\" (UniqueName: \"kubernetes.io/projected/0a759655-221d-4180-af64-e8bf585bcbdb-kube-api-access-scvwt\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.255921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.255819    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9846e224-aeeb-4612-bb4b-e66f8366c1dc-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.255921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.255828    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmcpm\" (UniqueName: \"kubernetes.io/projected/9846e224-aeeb-4612-bb4b-e66f8366c1dc-kube-api-access-bmcpm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:49.257542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.257305    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r" event={"ID":"9846e224-aeeb-4612-bb4b-e66f8366c1dc","Type":"ContainerDied","Data":"2b9a15e942c4f25f8e6add1082e371fa473460d80b342ef38866639fd8f432da"}
Oct 31 16:34:49.257542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.257429    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"
Oct 31 16:34:49.336796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.336767    4519 scope.go:122] "RemoveContainer" containerID="7587a643026f6ca5459d9efed901916005d9d41444cea0c583cd10212c8fb2c9"
Oct 31 16:34:49.442490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.439570    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 16:34:49.442490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.439635    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 16:34:49.448970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.448920    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"]
Oct 31 16:34:49.540651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.540546    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mb94r"]
Oct 31 16:34:49.545628 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.545585    4519 patch_prober.go:29] interesting pod/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 16:34:49.545775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.545647    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 16:34:49.653674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.653639    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"]
Oct 31 16:34:49.661336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:49.661305    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-bqrp8"]
Oct 31 16:34:50.020603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.020566    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:34:50.020896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.020865    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 16:34:50.442974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.442879    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 16:34:50.443582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.443553    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 16:34:50.455254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.455225    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" path="/var/lib/kubelet/pods/0a759655-221d-4180-af64-e8bf585bcbdb/volumes"
Oct 31 16:34:50.455971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.455947    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" path="/var/lib/kubelet/pods/9846e224-aeeb-4612-bb4b-e66f8366c1dc/volumes"
Oct 31 16:34:50.498478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.498416    4519 patch_prober.go:29] interesting pod/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 16:34:50.498689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:50.498505    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 16:34:51.004827 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:51.004787    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:34:51.005132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:51.005099    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 16:34:51.449530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:51.449084    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 31 16:34:51.449530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:51.449152    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 31 16:34:51.522572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:51.509389    4519 patch_prober.go:29] interesting pod/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 16:34:51.522572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:51.509467    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 16:34:52.006081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.006045    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:34:52.006381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.006350    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:34:52.007047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.007028    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:34:52.444604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.444533    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:34:52.445747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.445724    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:34:52.502788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.502763    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:34:52.503734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:52.503715    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:34:53.834126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834092    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"]
Oct 31 16:34:53.834847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834829    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy"
Oct 31 16:34:53.834927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834848    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy"
Oct 31 16:34:53.834927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834857    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy"
Oct 31 16:34:53.834927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834862    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy"
Oct 31 16:34:53.834927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834875    4519 cpu_manager.go:414] "containerMap: removing container" podUID="290b5587-0512-44c5-8e6a-00634ee620bb" containerName="revoke-keys"
Oct 31 16:34:53.834927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.834880    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290b5587-0512-44c5-8e6a-00634ee620bb" containerName="revoke-keys"
Oct 31 16:34:53.835143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.835043    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a759655-221d-4180-af64-e8bf585bcbdb" containerName="istio-proxy"
Oct 31 16:34:53.835143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.835054    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9846e224-aeeb-4612-bb4b-e66f8366c1dc" containerName="istio-proxy"
Oct 31 16:34:53.835143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.835061    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="290b5587-0512-44c5-8e6a-00634ee620bb" containerName="revoke-keys"
Oct 31 16:34:53.840526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.840507    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:53.890048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.890009    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:34:53.890322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.890296    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:34:53.890322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.890296    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:34:53.890525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.890370    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:34:53.890570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.890518    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"]
Oct 31 16:34:53.991580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.991549    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mf5d\" (UniqueName: \"kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:53.991754 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.991613    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:53.991920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:53.991898    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.005629 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:34:54.005595    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:34:54.093533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.093408    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.093692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.093573    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mf5d\" (UniqueName: \"kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.093692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.093615    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.094171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.094150    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.095992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.095968    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.147972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.147938    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mf5d\" (UniqueName: \"kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.154223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.154200    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:54.412706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.412585    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"]
Oct 31 16:34:54.413138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.412836    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy" containerID="cri-o://60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2" gracePeriod=30
Oct 31 16:34:54.417499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.417474    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"]
Oct 31 16:34:54.417665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.417646    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" containerID="cri-o://ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749" gracePeriod=30
Oct 31 16:34:54.548041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:54.545678    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"]
Oct 31 16:34:54.548245 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:34:54.548216    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9465907_2772_48ca_beb0_e598129ee1ce.slice/crio-948e7ea26ffcaab73d4d377ce1085c025d139f928248118c1a3615c517c48e05 WatchSource:0}: Error finding container 948e7ea26ffcaab73d4d377ce1085c025d139f928248118c1a3615c517c48e05: Status 404 returned error can't find the container with id 948e7ea26ffcaab73d4d377ce1085c025d139f928248118c1a3615c517c48e05
Oct 31 16:34:55.360146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:55.360104    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7" event={"ID":"d9465907-2772-48ca-beb0-e598129ee1ce","Type":"ContainerStarted","Data":"86aa5647919f1df6eabba0dfc29a65a02879a8c63211a75e4a0922faf8ed1821"}
Oct 31 16:34:55.360656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:55.360160    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7" event={"ID":"d9465907-2772-48ca-beb0-e598129ee1ce","Type":"ContainerStarted","Data":"948e7ea26ffcaab73d4d377ce1085c025d139f928248118c1a3615c517c48e05"}
Oct 31 16:34:55.470987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:55.470922    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7" podStartSLOduration=2.470903808 podStartE2EDuration="2.470903808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:55.466923752 +0000 UTC m=+1939.166853062" watchObservedRunningTime="2026-10-31 16:34:55.470903808 +0000 UTC m=+1939.170833121"
Oct 31 16:34:56.368534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:56.368498    4519 generic.go:356] "Generic (PLEG): container finished" podID="d9465907-2772-48ca-beb0-e598129ee1ce" containerID="86aa5647919f1df6eabba0dfc29a65a02879a8c63211a75e4a0922faf8ed1821" exitCode=0
Oct 31 16:34:56.368534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:56.368538    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7" event={"ID":"d9465907-2772-48ca-beb0-e598129ee1ce","Type":"ContainerDied","Data":"86aa5647919f1df6eabba0dfc29a65a02879a8c63211a75e4a0922faf8ed1821"}
Oct 31 16:34:57.151079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.151041    4519 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:34:57.151271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.151108    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:34:57.753822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.753761    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:57.883497 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.883448    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca\") pod \"d9465907-2772-48ca-beb0-e598129ee1ce\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") "
Oct 31 16:34:57.883657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.883513    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d\" (UniqueName: \"kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d\") pod \"d9465907-2772-48ca-beb0-e598129ee1ce\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") "
Oct 31 16:34:57.883657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.883558    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca\") pod \"d9465907-2772-48ca-beb0-e598129ee1ce\" (UID: \"d9465907-2772-48ca-beb0-e598129ee1ce\") "
Oct 31 16:34:57.883855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.883830    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca" pod "d9465907-2772-48ca-beb0-e598129ee1ce" (UID: "d9465907-2772-48ca-beb0-e598129ee1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:57.884736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.884709    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:57.885854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.885825    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca" pod "d9465907-2772-48ca-beb0-e598129ee1ce" (UID: "d9465907-2772-48ca-beb0-e598129ee1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:34:57.898523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.898497    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d" pod "d9465907-2772-48ca-beb0-e598129ee1ce" (UID: "d9465907-2772-48ca-beb0-e598129ee1ce"). InnerVolumeSpecName "kube-api-access-9mf5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:57.986039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.985999    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mf5d\" (UniqueName: \"kubernetes.io/projected/d9465907-2772-48ca-beb0-e598129ee1ce-kube-api-access-9mf5d\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:57.986039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:57.986039    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9465907-2772-48ca-beb0-e598129ee1ce-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:58.386081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.386057    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"
Oct 31 16:34:58.386256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.386048    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7" event={"ID":"d9465907-2772-48ca-beb0-e598129ee1ce","Type":"ContainerDied","Data":"948e7ea26ffcaab73d4d377ce1085c025d139f928248118c1a3615c517c48e05"}
Oct 31 16:34:58.386256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.386195    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948e7ea26ffcaab73d4d377ce1085c025d139f928248118c1a3615c517c48e05"
Oct 31 16:34:58.807130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.807089    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"]
Oct 31 16:34:58.808159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.808134    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d9465907-2772-48ca-beb0-e598129ee1ce" containerName="revoke-keys"
Oct 31 16:34:58.808159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.808157    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9465907-2772-48ca-beb0-e598129ee1ce" containerName="revoke-keys"
Oct 31 16:34:58.808389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.808320    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9465907-2772-48ca-beb0-e598129ee1ce" containerName="revoke-keys"
Oct 31 16:34:58.814639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.814605    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:58.864575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.864546    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:34:58.864856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.864604    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:34:58.888571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.888537    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"]
Oct 31 16:34:58.899873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.899839    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:34:58.899873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.899843    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:34:58.901113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.901087    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:58.901431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.901406    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2ln\" (UniqueName: \"kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:58.901598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:58.901495    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.058535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.057044    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2ln\" (UniqueName: \"kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.058535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.057097    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.058535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.057263    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.058535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.058157    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.105587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.105515    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.138853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.138186    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2ln\" (UniqueName: \"kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.157035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.157004    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:34:59.668007 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:34:59.667924    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"]
Oct 31 16:34:59.697591 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:34:59.693147    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c9c006_67c5_4e45_b613_13cb7e63e152.slice/crio-8bb4285b6e70ea43b7ab97c2fc0c2097df33895d3ed21077c8ca1e8504356220 WatchSource:0}: Error finding container 8bb4285b6e70ea43b7ab97c2fc0c2097df33895d3ed21077c8ca1e8504356220: Status 404 returned error can't find the container with id 8bb4285b6e70ea43b7ab97c2fc0c2097df33895d3ed21077c8ca1e8504356220
Oct 31 16:35:00.013245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.013216    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:35:00.027711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.027683    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:35:00.112265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112232    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112275    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.112504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112301    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.112504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112317    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112405    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.112504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112491    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112525    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112593    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112627    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112643    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112678    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112714    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.112792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112740    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.113123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112880    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.113123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112906    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.113123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112916    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:00.113123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.112915    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.113123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113015    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.113123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113052    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.113469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.113469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113171    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.113469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113191    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket\") pod \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\" (UID: \"bf6bc8de-16c6-4fd1-aba9-92773f73f00d\") "
Oct 31 16:35:00.113469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113191    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.113469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113231    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token\") pod \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\" (UID: \"6e574cac-dbd0-4e63-9a50-396cf7bde4fb\") "
Oct 31 16:35:00.113469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.113429    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.114171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114143    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:00.114296 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114266    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.114678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114652    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114686    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114701    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114714    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114728    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114744    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114758    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.114776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114771    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.115149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114785    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.115149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.114865    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.116121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116024    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "kube-api-access-wzgxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:00.116121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116065    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.116121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116081    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "kube-api-access-5dk22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:00.116523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116493    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:00.116523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116511    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:35:00.116694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116558    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:35:00.116694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116643    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token" pod "6e574cac-dbd0-4e63-9a50-396cf7bde4fb" (UID: "6e574cac-dbd0-4e63-9a50-396cf7bde4fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:00.116886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.116870    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token" pod "bf6bc8de-16c6-4fd1-aba9-92773f73f00d" (UID: "bf6bc8de-16c6-4fd1-aba9-92773f73f00d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:00.216493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216441    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216491    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216507    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216519    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216532    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dk22\" (UniqueName: \"kubernetes.io/projected/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-kube-api-access-5dk22\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216544    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgxm\" (UniqueName: \"kubernetes.io/projected/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-kube-api-access-wzgxm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216555    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216570    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e574cac-dbd0-4e63-9a50-396cf7bde4fb-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.216724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.216582    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6bc8de-16c6-4fd1-aba9-92773f73f00d-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:00.409533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.409491    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s" event={"ID":"53c9c006-67c5-4e45-b613-13cb7e63e152","Type":"ContainerStarted","Data":"68c0c8902830d248fd2f5b5ed91735a95940699c89ed99d6fce674f5767b8a9c"}
Oct 31 16:35:00.409744 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.409597    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s" event={"ID":"53c9c006-67c5-4e45-b613-13cb7e63e152","Type":"ContainerStarted","Data":"8bb4285b6e70ea43b7ab97c2fc0c2097df33895d3ed21077c8ca1e8504356220"}
Oct 31 16:35:00.411415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.411384    4519 generic.go:356] "Generic (PLEG): container finished" podID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerID="ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749" exitCode=0
Oct 31 16:35:00.411557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.411421    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" event={"ID":"6e574cac-dbd0-4e63-9a50-396cf7bde4fb","Type":"ContainerDied","Data":"ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749"}
Oct 31 16:35:00.411557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.411517    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg" event={"ID":"6e574cac-dbd0-4e63-9a50-396cf7bde4fb","Type":"ContainerDied","Data":"95775183eb5aa46f38447f090cb1e94f91b297b73d6063b7f8e03ac5aa502968"}
Oct 31 16:35:00.411557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.411538    4519 scope.go:122] "RemoveContainer" containerID="ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749"
Oct 31 16:35:00.411740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.411429    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"
Oct 31 16:35:00.413196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.413178    4519 generic.go:356] "Generic (PLEG): container finished" podID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerID="60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2" exitCode=0
Oct 31 16:35:00.413301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.413227    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"
Oct 31 16:35:00.413301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.413269    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" event={"ID":"bf6bc8de-16c6-4fd1-aba9-92773f73f00d","Type":"ContainerDied","Data":"60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2"}
Oct 31 16:35:00.413407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.413317    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq" event={"ID":"bf6bc8de-16c6-4fd1-aba9-92773f73f00d","Type":"ContainerDied","Data":"3e61a54d6406e2613b74eeb22ef2ee4782ae596944e2e8daf079e3f1b4a4356d"}
Oct 31 16:35:00.423839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.423822    4519 scope.go:122] "RemoveContainer" containerID="ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749"
Oct 31 16:35:00.424152 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:35:00.424132    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749\": container with ID starting with ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749 not found: ID does not exist" containerID="ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749"
Oct 31 16:35:00.424233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.424159    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749"} err="failed to get container status \"ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749\": rpc error: code = NotFound desc = could not find container \"ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749\": container with ID starting with ff409113957c05eae11cf88bb75b9b970730ef8a4c1f1a6a86b2b2a54866c749 not found: ID does not exist"
Oct 31 16:35:00.424233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.424182    4519 scope.go:122] "RemoveContainer" containerID="60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2"
Oct 31 16:35:00.433758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.433737    4519 scope.go:122] "RemoveContainer" containerID="60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2"
Oct 31 16:35:00.434026 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:35:00.434003    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2\": container with ID starting with 60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2 not found: ID does not exist" containerID="60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2"
Oct 31 16:35:00.434100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.434032    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2"} err="failed to get container status \"60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2\": rpc error: code = NotFound desc = could not find container \"60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2\": container with ID starting with 60086d2f12fad752a220425b5b82e6f69781bc4f4c62e3a6e5b8dce3b072fbd2 not found: ID does not exist"
Oct 31 16:35:00.517716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.517622    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s" podStartSLOduration=2.5176059840000002 podStartE2EDuration="2.517605984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:00.516889711 +0000 UTC m=+1944.216819022" watchObservedRunningTime="2026-10-31 16:35:00.517605984 +0000 UTC m=+1944.217535274"
Oct 31 16:35:00.643915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.643882    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"]
Oct 31 16:35:00.648481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.648445    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p58mg"]
Oct 31 16:35:00.754320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.754282    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"]
Oct 31 16:35:00.758388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:00.758359    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-jqtwq"]
Oct 31 16:35:01.425403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:01.425374    4519 generic.go:356] "Generic (PLEG): container finished" podID="53c9c006-67c5-4e45-b613-13cb7e63e152" containerID="68c0c8902830d248fd2f5b5ed91735a95940699c89ed99d6fce674f5767b8a9c" exitCode=0
Oct 31 16:35:01.425798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:01.425450    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s" event={"ID":"53c9c006-67c5-4e45-b613-13cb7e63e152","Type":"ContainerDied","Data":"68c0c8902830d248fd2f5b5ed91735a95940699c89ed99d6fce674f5767b8a9c"}
Oct 31 16:35:02.442872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.442840    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" path="/var/lib/kubelet/pods/6e574cac-dbd0-4e63-9a50-396cf7bde4fb/volumes"
Oct 31 16:35:02.443613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.443590    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" path="/var/lib/kubelet/pods/bf6bc8de-16c6-4fd1-aba9-92773f73f00d/volumes"
Oct 31 16:35:02.716755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.716726    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:35:02.756757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.756729    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln\" (UniqueName: \"kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln\") pod \"53c9c006-67c5-4e45-b613-13cb7e63e152\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") "
Oct 31 16:35:02.756914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.756829    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca\") pod \"53c9c006-67c5-4e45-b613-13cb7e63e152\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") "
Oct 31 16:35:02.756954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.756938    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca\") pod \"53c9c006-67c5-4e45-b613-13cb7e63e152\" (UID: \"53c9c006-67c5-4e45-b613-13cb7e63e152\") "
Oct 31 16:35:02.757309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.757285    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca" pod "53c9c006-67c5-4e45-b613-13cb7e63e152" (UID: "53c9c006-67c5-4e45-b613-13cb7e63e152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:02.757524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.757504    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:02.758859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.758837    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca" pod "53c9c006-67c5-4e45-b613-13cb7e63e152" (UID: "53c9c006-67c5-4e45-b613-13cb7e63e152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:02.758964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.758943    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln" pod "53c9c006-67c5-4e45-b613-13cb7e63e152" (UID: "53c9c006-67c5-4e45-b613-13cb7e63e152"). InnerVolumeSpecName "kube-api-access-7b2ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:02.859415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.859386    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c9c006-67c5-4e45-b613-13cb7e63e152-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:02.859415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:02.859415    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b2ln\" (UniqueName: \"kubernetes.io/projected/53c9c006-67c5-4e45-b613-13cb7e63e152-kube-api-access-7b2ln\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:03.444177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:03.444149    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"
Oct 31 16:35:03.444177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:03.444162    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s" event={"ID":"53c9c006-67c5-4e45-b613-13cb7e63e152","Type":"ContainerDied","Data":"8bb4285b6e70ea43b7ab97c2fc0c2097df33895d3ed21077c8ca1e8504356220"}
Oct 31 16:35:03.444631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:03.444213    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb4285b6e70ea43b7ab97c2fc0c2097df33895d3ed21077c8ca1e8504356220"
Oct 31 16:35:05.009154 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009116    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"]
Oct 31 16:35:05.009842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009828    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy"
Oct 31 16:35:05.009842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009843    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy"
Oct 31 16:35:05.009912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009886    4519 cpu_manager.go:414] "containerMap: removing container" podUID="53c9c006-67c5-4e45-b613-13cb7e63e152" containerName="revoke-keys"
Oct 31 16:35:05.009912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009893    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c9c006-67c5-4e45-b613-13cb7e63e152" containerName="revoke-keys"
Oct 31 16:35:05.009912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009901    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy"
Oct 31 16:35:05.009912 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.009906    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy"
Oct 31 16:35:05.010049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.010017    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c9c006-67c5-4e45-b613-13cb7e63e152" containerName="revoke-keys"
Oct 31 16:35:05.010049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.010028    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6bc8de-16c6-4fd1-aba9-92773f73f00d" containerName="istio-proxy"
Oct 31 16:35:05.010049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.010040    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e574cac-dbd0-4e63-9a50-396cf7bde4fb" containerName="istio-proxy"
Oct 31 16:35:05.014040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.014018    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.094480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.093639    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"]
Oct 31 16:35:05.109330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.109298    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:35:05.109620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.109321    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:35:05.109620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.109325    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:35:05.109787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.109333    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:35:05.116211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.116172    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.116372 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.116344    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8pzl\" (UniqueName: \"kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.116688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.116670    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.218383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.218347    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.218594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.218468    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.218594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.218534    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8pzl\" (UniqueName: \"kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.219106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.219081    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.220878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.220857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.287970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.287890    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8pzl\" (UniqueName: \"kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl\") pod \"maas-api-revoke-keys-e2e-api-a-dac4fa-59phz\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.332720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.332688    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:05.715271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:05.715241    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"]
Oct 31 16:35:05.719273 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:35:05.719239    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc977453d_86d9_4d01_bcda_f5996605e517.slice/crio-34c2239e1b42719a495d43b4922b4b284763feac1ad42e8d37f934462fc6bb23 WatchSource:0}: Error finding container 34c2239e1b42719a495d43b4922b4b284763feac1ad42e8d37f934462fc6bb23: Status 404 returned error can't find the container with id 34c2239e1b42719a495d43b4922b4b284763feac1ad42e8d37f934462fc6bb23
Oct 31 16:35:06.475320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:06.475282    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz" event={"ID":"c977453d-86d9-4d01-bcda-f5996605e517","Type":"ContainerStarted","Data":"9b390d78022c325904e3c454e5bc3c0118224f6a8a1d41b6f17e7f09543410b4"}
Oct 31 16:35:06.475743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:06.475333    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz" event={"ID":"c977453d-86d9-4d01-bcda-f5996605e517","Type":"ContainerStarted","Data":"34c2239e1b42719a495d43b4922b4b284763feac1ad42e8d37f934462fc6bb23"}
Oct 31 16:35:06.584561 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:06.584499    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz" podStartSLOduration=2.584472611 podStartE2EDuration="2.584472611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:06.581827914 +0000 UTC m=+1950.281757224" watchObservedRunningTime="2026-10-31 16:35:06.584472611 +0000 UTC m=+1950.284401923"
Oct 31 16:35:06.612609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:06.612576    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:35:06.612787 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:06.612632    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:35:07.485362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:07.485321    4519 generic.go:356] "Generic (PLEG): container finished" podID="c977453d-86d9-4d01-bcda-f5996605e517" containerID="9b390d78022c325904e3c454e5bc3c0118224f6a8a1d41b6f17e7f09543410b4" exitCode=0
Oct 31 16:35:07.485845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:07.485393    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz" event={"ID":"c977453d-86d9-4d01-bcda-f5996605e517","Type":"ContainerDied","Data":"9b390d78022c325904e3c454e5bc3c0118224f6a8a1d41b6f17e7f09543410b4"}
Oct 31 16:35:08.671802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.671770    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:08.749731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.749651    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl\" (UniqueName: \"kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl\") pod \"c977453d-86d9-4d01-bcda-f5996605e517\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") "
Oct 31 16:35:08.749901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.749756    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca\") pod \"c977453d-86d9-4d01-bcda-f5996605e517\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") "
Oct 31 16:35:08.750103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.750077    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca\") pod \"c977453d-86d9-4d01-bcda-f5996605e517\" (UID: \"c977453d-86d9-4d01-bcda-f5996605e517\") "
Oct 31 16:35:08.750247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.750125    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca" pod "c977453d-86d9-4d01-bcda-f5996605e517" (UID: "c977453d-86d9-4d01-bcda-f5996605e517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:08.750838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.750814    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:08.751885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.751863    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl" pod "c977453d-86d9-4d01-bcda-f5996605e517" (UID: "c977453d-86d9-4d01-bcda-f5996605e517"). InnerVolumeSpecName "kube-api-access-g8pzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:08.752051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.752037    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca" pod "c977453d-86d9-4d01-bcda-f5996605e517" (UID: "c977453d-86d9-4d01-bcda-f5996605e517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:08.852164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.852124    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8pzl\" (UniqueName: \"kubernetes.io/projected/c977453d-86d9-4d01-bcda-f5996605e517-kube-api-access-g8pzl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:08.852164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:08.852156    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c977453d-86d9-4d01-bcda-f5996605e517-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:09.502889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.502863    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"
Oct 31 16:35:09.502889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.502888    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz" event={"ID":"c977453d-86d9-4d01-bcda-f5996605e517","Type":"ContainerDied","Data":"34c2239e1b42719a495d43b4922b4b284763feac1ad42e8d37f934462fc6bb23"}
Oct 31 16:35:09.503178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.502942    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34c2239e1b42719a495d43b4922b4b284763feac1ad42e8d37f934462fc6bb23"
Oct 31 16:35:09.504777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.504750    4519 generic.go:356] "Generic (PLEG): container finished" podID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerID="1adf9c6c1dfd7eb4a25219c6f60dc05c95d4136faf953e407d40645884253e10" exitCode=0
Oct 31 16:35:09.504894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.504815    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" event={"ID":"3650035e-9a85-4eee-8dd4-a2bab7815442","Type":"ContainerDied","Data":"1adf9c6c1dfd7eb4a25219c6f60dc05c95d4136faf953e407d40645884253e10"}
Oct 31 16:35:09.505338 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.505321    4519 scope.go:122] "RemoveContainer" containerID="1adf9c6c1dfd7eb4a25219c6f60dc05c95d4136faf953e407d40645884253e10"
Oct 31 16:35:09.507448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.507416    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:09.507555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.507526    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:09.507595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:09.507578    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:10.514215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:10.514176    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" event={"ID":"3650035e-9a85-4eee-8dd4-a2bab7815442","Type":"ContainerStarted","Data":"1bb8aeef58376a27e72351d4901b4fce1bc43db98a26c277698ad66e68615230"}
Oct 31 16:35:11.090650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:11.090610    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:35:11.090927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:11.090890    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 16:35:11.091086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:11.090944    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 16:35:12.090923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:12.090874    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 16:35:12.091425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:12.090947    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 16:35:13.090481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:13.089831    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 16:35:13.090481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:13.089897    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 16:35:14.090879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:14.090845    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 16:35:14.091362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:14.091333    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 16:35:15.089861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:15.089822    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 16:35:15.090157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:15.090128    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 16:35:16.097479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.091887    4519 patch_prober.go:29] interesting pod/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 16:35:16.097479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.091965    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 16:35:16.225622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.225065    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"]
Oct 31 16:35:16.237732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.231993    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c977453d-86d9-4d01-bcda-f5996605e517" containerName="revoke-keys"
Oct 31 16:35:16.237732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.237543    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977453d-86d9-4d01-bcda-f5996605e517" containerName="revoke-keys"
Oct 31 16:35:16.237991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.237816    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977453d-86d9-4d01-bcda-f5996605e517" containerName="revoke-keys"
Oct 31 16:35:16.244037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.244010    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.286425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.286381    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"]
Oct 31 16:35:16.293990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.293963    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:35:16.294422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.294292    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:35:16.294635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.294355    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:35:16.324544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.324515    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:35:16.344418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.344374    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfnw\" (UniqueName: \"kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.344678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.344658    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.344777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.344745    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.445681 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.445596    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfnw\" (UniqueName: \"kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.445869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.445766    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.445869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.445821    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.446403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.446380    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.448221 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.448193    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.499175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.499138    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfnw\" (UniqueName: \"kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw\") pod \"maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.567819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.567776    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:16.950149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:16.950115    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"]
Oct 31 16:35:16.954310 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:35:16.954283    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffef1219_505e_4503_b42e_9ed7996c25ff.slice/crio-5af1369fb54d51072911fc392f0faccc3a57b61c29fbdaf69d2b2534593ba180 WatchSource:0}: Error finding container 5af1369fb54d51072911fc392f0faccc3a57b61c29fbdaf69d2b2534593ba180: Status 404 returned error can't find the container with id 5af1369fb54d51072911fc392f0faccc3a57b61c29fbdaf69d2b2534593ba180
Oct 31 16:35:17.093584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:17.093552    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:35:17.093985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:17.093961    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:35:17.094725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:17.094707    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:35:17.643016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:17.642972    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f" event={"ID":"ffef1219-505e-4503-b42e-9ed7996c25ff","Type":"ContainerStarted","Data":"fe4aa5b39cac1b832ff69c7a02c3a509d51b216643cae8250cce1b528c007529"}
Oct 31 16:35:17.643488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:17.643039    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f" event={"ID":"ffef1219-505e-4503-b42e-9ed7996c25ff","Type":"ContainerStarted","Data":"5af1369fb54d51072911fc392f0faccc3a57b61c29fbdaf69d2b2534593ba180"}
Oct 31 16:35:17.756119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:17.756062    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f" podStartSLOduration=1.75604974 podStartE2EDuration="1.75604974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:17.751914612 +0000 UTC m=+1961.451843923" watchObservedRunningTime="2026-10-31 16:35:17.75604974 +0000 UTC m=+1961.455979050"
Oct 31 16:35:18.652130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:18.652097    4519 generic.go:356] "Generic (PLEG): container finished" podID="ffef1219-505e-4503-b42e-9ed7996c25ff" containerID="fe4aa5b39cac1b832ff69c7a02c3a509d51b216643cae8250cce1b528c007529" exitCode=0
Oct 31 16:35:18.652540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:18.652191    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f" event={"ID":"ffef1219-505e-4503-b42e-9ed7996c25ff","Type":"ContainerDied","Data":"fe4aa5b39cac1b832ff69c7a02c3a509d51b216643cae8250cce1b528c007529"}
Oct 31 16:35:19.840953 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.840926    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:19.951648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.951571    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw\" (UniqueName: \"kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw\") pod \"ffef1219-505e-4503-b42e-9ed7996c25ff\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") "
Oct 31 16:35:19.951648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.951631    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca\") pod \"ffef1219-505e-4503-b42e-9ed7996c25ff\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") "
Oct 31 16:35:19.951868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.951673    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca\") pod \"ffef1219-505e-4503-b42e-9ed7996c25ff\" (UID: \"ffef1219-505e-4503-b42e-9ed7996c25ff\") "
Oct 31 16:35:19.952594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.952564    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca" pod "ffef1219-505e-4503-b42e-9ed7996c25ff" (UID: "ffef1219-505e-4503-b42e-9ed7996c25ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:19.954905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.954879    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw" pod "ffef1219-505e-4503-b42e-9ed7996c25ff" (UID: "ffef1219-505e-4503-b42e-9ed7996c25ff"). InnerVolumeSpecName "kube-api-access-4vfnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:19.955119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:19.955082    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca" pod "ffef1219-505e-4503-b42e-9ed7996c25ff" (UID: "ffef1219-505e-4503-b42e-9ed7996c25ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:20.053282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:20.053251    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfnw\" (UniqueName: \"kubernetes.io/projected/ffef1219-505e-4503-b42e-9ed7996c25ff-kube-api-access-4vfnw\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:20.053282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:20.053277    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:20.053282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:20.053289    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffef1219-505e-4503-b42e-9ed7996c25ff-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:20.670748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:20.670717    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"
Oct 31 16:35:20.670947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:20.670753    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f" event={"ID":"ffef1219-505e-4503-b42e-9ed7996c25ff","Type":"ContainerDied","Data":"5af1369fb54d51072911fc392f0faccc3a57b61c29fbdaf69d2b2534593ba180"}
Oct 31 16:35:20.670947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:20.670799    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af1369fb54d51072911fc392f0faccc3a57b61c29fbdaf69d2b2534593ba180"
Oct 31 16:35:24.122234 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:35:24.122192    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:35:26.440937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:26.440908    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"]
Oct 31 16:35:26.481815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:26.481783    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e977aa-fx5wv"]
Oct 31 16:35:28.440527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:28.440494    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0948af8f-66a8-48e2-ad44-1e1cf333cd0c" path="/var/lib/kubelet/pods/0948af8f-66a8-48e2-ad44-1e1cf333cd0c/volumes"
Oct 31 16:35:38.475145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:38.475114    4519 scope.go:122] "RemoveContainer" containerID="4453d3b318423c36e492b45beecc4e67d99d4b8d81b85b836b192d0caf3ce2e2"
Oct 31 16:35:44.071584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:44.071549    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"]
Oct 31 16:35:44.118290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:44.118253    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdc828-ztx6f"]
Oct 31 16:35:44.451900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:44.451825    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffef1219-505e-4503-b42e-9ed7996c25ff" path="/var/lib/kubelet/pods/ffef1219-505e-4503-b42e-9ed7996c25ff/volumes"
Oct 31 16:35:46.772778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:46.772745    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:35:50.535336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:50.535293    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"]
Oct 31 16:35:50.535957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:50.535570    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" containerID="cri-o://a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606" gracePeriod=30
Oct 31 16:35:51.643548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.643515    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"]
Oct 31 16:35:51.644238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.644221    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ffef1219-505e-4503-b42e-9ed7996c25ff" containerName="revoke-keys"
Oct 31 16:35:51.644286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.644240    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffef1219-505e-4503-b42e-9ed7996c25ff" containerName="revoke-keys"
Oct 31 16:35:51.644421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.644411    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffef1219-505e-4503-b42e-9ed7996c25ff" containerName="revoke-keys"
Oct 31 16:35:51.649731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.649713    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.703927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.703893    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"]
Oct 31 16:35:51.745751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.745721    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:35:51.745751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.745742    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:35:51.745967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.745743    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:35:51.747229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.747213    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:35:51.773859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.773827    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2m7x\" (UniqueName: \"kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.774050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.773975    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.774050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.774034    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.875275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.875239    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2m7x\" (UniqueName: \"kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.875451 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.875341    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.875451 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.875361    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.876054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.876023    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.877761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.877733    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.934330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.934253    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2m7x\" (UniqueName: \"kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x\") pod \"maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:51.971413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:51.971387    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:52.355552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:52.355520    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"]
Oct 31 16:35:52.359945 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:35:52.359909    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2318fc10_8d6b_438f_83df_e93d532b7df1.slice/crio-3383bab1d6cff55a8db0309ea738d1ea94859497547ef82d46508ac7a3bf3663 WatchSource:0}: Error finding container 3383bab1d6cff55a8db0309ea738d1ea94859497547ef82d46508ac7a3bf3663: Status 404 returned error can't find the container with id 3383bab1d6cff55a8db0309ea738d1ea94859497547ef82d46508ac7a3bf3663
Oct 31 16:35:52.447289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:52.447257    4519 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:35:52.447488 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:52.447309    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:35:52.943524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:52.943490    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2" event={"ID":"2318fc10-8d6b-438f-83df-e93d532b7df1","Type":"ContainerStarted","Data":"cd5ab88086d1b9bd4d4e7592b1a224ec164e645594d0774e7eb2ef86d4d1cf3e"}
Oct 31 16:35:52.943936 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:52.943550    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2" event={"ID":"2318fc10-8d6b-438f-83df-e93d532b7df1","Type":"ContainerStarted","Data":"3383bab1d6cff55a8db0309ea738d1ea94859497547ef82d46508ac7a3bf3663"}
Oct 31 16:35:53.055097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:53.054993    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2" podStartSLOduration=2.054980338 podStartE2EDuration="2.054980338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:53.05327115 +0000 UTC m=+1996.753200461" watchObservedRunningTime="2026-10-31 16:35:53.054980338 +0000 UTC m=+1996.754909650"
Oct 31 16:35:54.002543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:53.991690    4519 generic.go:356] "Generic (PLEG): container finished" podID="2318fc10-8d6b-438f-83df-e93d532b7df1" containerID="cd5ab88086d1b9bd4d4e7592b1a224ec164e645594d0774e7eb2ef86d4d1cf3e" exitCode=0
Oct 31 16:35:54.002543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:53.991917    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2" event={"ID":"2318fc10-8d6b-438f-83df-e93d532b7df1","Type":"ContainerDied","Data":"cd5ab88086d1b9bd4d4e7592b1a224ec164e645594d0774e7eb2ef86d4d1cf3e"}
Oct 31 16:35:54.140939 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:35:54.140901    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:35:55.180879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.180856    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:55.225141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.225109    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"]
Oct 31 16:35:55.272234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.272146    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-4lcck"]
Oct 31 16:35:55.330702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.330664    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca\") pod \"2318fc10-8d6b-438f-83df-e93d532b7df1\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") "
Oct 31 16:35:55.331334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.331054    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca\") pod \"2318fc10-8d6b-438f-83df-e93d532b7df1\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") "
Oct 31 16:35:55.331334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.331329    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x\" (UniqueName: \"kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x\") pod \"2318fc10-8d6b-438f-83df-e93d532b7df1\" (UID: \"2318fc10-8d6b-438f-83df-e93d532b7df1\") "
Oct 31 16:35:55.331638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.331614    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca" pod "2318fc10-8d6b-438f-83df-e93d532b7df1" (UID: "2318fc10-8d6b-438f-83df-e93d532b7df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:55.332198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.332125    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.333736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.333574    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca" pod "2318fc10-8d6b-438f-83df-e93d532b7df1" (UID: "2318fc10-8d6b-438f-83df-e93d532b7df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:55.346674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.346644    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x" pod "2318fc10-8d6b-438f-83df-e93d532b7df1" (UID: "2318fc10-8d6b-438f-83df-e93d532b7df1"). InnerVolumeSpecName "kube-api-access-k2m7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:55.434196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.434158    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2318fc10-8d6b-438f-83df-e93d532b7df1-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.434196 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.434189    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2m7x\" (UniqueName: \"kubernetes.io/projected/2318fc10-8d6b-438f-83df-e93d532b7df1-kube-api-access-k2m7x\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.819650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.819619    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:35:55.941438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941402    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.941642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941475    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.941642 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941549    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.941741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941647    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.941741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941726    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.941741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941725    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.941865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941831    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.941987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.941943    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.942159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.942093    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.942223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.942156    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo\") pod \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\" (UID: \"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a\") "
Oct 31 16:35:55.942789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.942098    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:55.943186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.943141    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.943186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.943162    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.943186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.942339    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.943186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.942547    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.943186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.942714    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.944922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.944895    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.945192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.945165    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:35:55.946082 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.946053    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "kube-api-access-dv76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:55.946353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:55.946318    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token" pod "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" (UID: "2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:56.012562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.012524    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2" event={"ID":"2318fc10-8d6b-438f-83df-e93d532b7df1","Type":"ContainerDied","Data":"3383bab1d6cff55a8db0309ea738d1ea94859497547ef82d46508ac7a3bf3663"}
Oct 31 16:35:56.012816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.012617    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3383bab1d6cff55a8db0309ea738d1ea94859497547ef82d46508ac7a3bf3663"
Oct 31 16:35:56.012816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.012681    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"
Oct 31 16:35:56.024704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.024639    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"
Oct 31 16:35:56.024806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.024779    4519 generic.go:356] "Generic (PLEG): container finished" podID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerID="a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606" exitCode=0
Oct 31 16:35:56.025012 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.024833    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" event={"ID":"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a","Type":"ContainerDied","Data":"a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606"}
Oct 31 16:35:56.025121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.025047    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f" event={"ID":"2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a","Type":"ContainerDied","Data":"6d1938c6c6bb59fcf6d0b4325c9b338681400cba3674660fd72ab2d2d76839d8"}
Oct 31 16:35:56.025121 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.025080    4519 scope.go:122] "RemoveContainer" containerID="a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606"
Oct 31 16:35:56.036976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.036961    4519 scope.go:122] "RemoveContainer" containerID="fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"
Oct 31 16:35:56.044407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044383    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv76w\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-kube-api-access-dv76w\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.044534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044413    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.044534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044430    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.044534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044446    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.044534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044498    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.044534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044512    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.044534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.044526    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.046917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.046897    4519 scope.go:122] "RemoveContainer" containerID="a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606"
Oct 31 16:35:56.047186 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:35:56.047168    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606\": container with ID starting with a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606 not found: ID does not exist" containerID="a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606"
Oct 31 16:35:56.047264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.047198    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606"} err="failed to get container status \"a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606\": rpc error: code = NotFound desc = could not find container \"a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606\": container with ID starting with a976daff5a98446e2d51f8688c3d46426d1d4d536c277a91c5fa5b47167ff606 not found: ID does not exist"
Oct 31 16:35:56.047264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.047223    4519 scope.go:122] "RemoveContainer" containerID="fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"
Oct 31 16:35:56.047476 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:35:56.047440    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829\": container with ID starting with fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829 not found: ID does not exist" containerID="fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"
Oct 31 16:35:56.047560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.047483    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829"} err="failed to get container status \"fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829\": rpc error: code = NotFound desc = could not find container \"fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829\": container with ID starting with fffda3adb3a56feac1b11511e8bbe5d12f86095a3a15b9edb1cfbdd73549a829 not found: ID does not exist"
Oct 31 16:35:56.236216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.236179    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"]
Oct 31 16:35:56.243562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.243530    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-grp6f"]
Oct 31 16:35:56.445641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.445558    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" path="/var/lib/kubelet/pods/2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a/volumes"
Oct 31 16:35:56.446099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:35:56.446081    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6635f83-c118-452d-9e8a-ddf372396c2a" path="/var/lib/kubelet/pods/a6635f83-c118-452d-9e8a-ddf372396c2a/volumes"
Oct 31 16:36:16.604382 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:16.604341    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"]
Oct 31 16:36:16.651222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:16.651182    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a2ae37-5ppw2"]
Oct 31 16:36:18.442353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:18.442266    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2318fc10-8d6b-438f-83df-e93d532b7df1" path="/var/lib/kubelet/pods/2318fc10-8d6b-438f-83df-e93d532b7df1/volumes"
Oct 31 16:36:23.141669 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:23.141627    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"]
Oct 31 16:36:23.142203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:23.141886    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy" containerID="cri-o://1bb8aeef58376a27e72351d4901b4fce1bc43db98a26c277698ad66e68615230" gracePeriod=30
Oct 31 16:36:24.005821 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:36:24.005784    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:36:25.361218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361180    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"]
Oct 31 16:36:25.361900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361885    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy"
Oct 31 16:36:25.361952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361901    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy"
Oct 31 16:36:25.361952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361911    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2318fc10-8d6b-438f-83df-e93d532b7df1" containerName="revoke-keys"
Oct 31 16:36:25.361952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361916    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2318fc10-8d6b-438f-83df-e93d532b7df1" containerName="revoke-keys"
Oct 31 16:36:25.362063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361957    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy"
Oct 31 16:36:25.362063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.361963    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy"
Oct 31 16:36:25.362131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.362079    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy"
Oct 31 16:36:25.362131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.362088    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2318fc10-8d6b-438f-83df-e93d532b7df1" containerName="revoke-keys"
Oct 31 16:36:25.366217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.366198    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.419443 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.419412    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"]
Oct 31 16:36:25.461390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.461362    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:36:25.461390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.461363    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:36:25.461624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.461396    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:36:25.463144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.463126    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:36:25.503023 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.502985    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.503165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.503097    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.503165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.503148    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmsb\" (UniqueName: \"kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.604674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.604643    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.604862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.604695    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.604862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.604730    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmsb\" (UniqueName: \"kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.605338 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.605313    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.607114 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.607089    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.658882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.658814    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmsb\" (UniqueName: \"kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:25.679859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:25.679836    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:26.055522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:26.055490    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"]
Oct 31 16:36:26.060195 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:36:26.060163    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be8ca25_8b9e_4166_9599_798196ac0bc7.slice/crio-00f79ed30466630c7677e9d459a50bf5fc13abb36b692a1f59cd128ff849c296 WatchSource:0}: Error finding container 00f79ed30466630c7677e9d459a50bf5fc13abb36b692a1f59cd128ff849c296: Status 404 returned error can't find the container with id 00f79ed30466630c7677e9d459a50bf5fc13abb36b692a1f59cd128ff849c296
Oct 31 16:36:26.394230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:26.394145    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk" event={"ID":"6be8ca25-8b9e-4166-9599-798196ac0bc7","Type":"ContainerStarted","Data":"00f79ed30466630c7677e9d459a50bf5fc13abb36b692a1f59cd128ff849c296"}
Oct 31 16:36:27.404904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:27.404818    4519 generic.go:356] "Generic (PLEG): container finished" podID="6be8ca25-8b9e-4166-9599-798196ac0bc7" containerID="349bf0d11d73d62cc5f7e45cd6926cdd826d641977aa9aaa5377ba0f129994ec" exitCode=0
Oct 31 16:36:27.405272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:27.404895    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk" event={"ID":"6be8ca25-8b9e-4166-9599-798196ac0bc7","Type":"ContainerDied","Data":"349bf0d11d73d62cc5f7e45cd6926cdd826d641977aa9aaa5377ba0f129994ec"}
Oct 31 16:36:28.416759 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.416675    4519 generic.go:356] "Generic (PLEG): container finished" podID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerID="1bb8aeef58376a27e72351d4901b4fce1bc43db98a26c277698ad66e68615230" exitCode=0
Oct 31 16:36:28.416759 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.416746    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" event={"ID":"3650035e-9a85-4eee-8dd4-a2bab7815442","Type":"ContainerDied","Data":"1bb8aeef58376a27e72351d4901b4fce1bc43db98a26c277698ad66e68615230"}
Oct 31 16:36:28.417175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.416802    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4" event={"ID":"3650035e-9a85-4eee-8dd4-a2bab7815442","Type":"ContainerDied","Data":"77af27a8505028ab20c14e55a89b3216fedbea8c3f41055e71deec85bd0b8ee2"}
Oct 31 16:36:28.417175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.416817    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77af27a8505028ab20c14e55a89b3216fedbea8c3f41055e71deec85bd0b8ee2"
Oct 31 16:36:28.417175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.416831    4519 scope.go:122] "RemoveContainer" containerID="1adf9c6c1dfd7eb4a25219c6f60dc05c95d4136faf953e407d40645884253e10"
Oct 31 16:36:28.427823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.427797    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:36:28.444735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.444706    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.444910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.444803    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.444910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.444857    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.444910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.444907    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.445122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445106    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.445162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445139    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.445223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445174    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.445223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445209    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.445329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445267    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token\") pod \"3650035e-9a85-4eee-8dd4-a2bab7815442\" (UID: \"3650035e-9a85-4eee-8dd4-a2bab7815442\") "
Oct 31 16:36:28.445393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445361    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:28.445514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445490    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:28.445686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445663    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:28.445769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.445686    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:28.446055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.446029    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:28.446343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.446321    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3650035e-9a85-4eee-8dd4-a2bab7815442-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.446427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.446349    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.446427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.446365    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.446427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.446379    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.446427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.446393    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.447369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.447349    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:28.447446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.447402    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "kube-api-access-9bb98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:28.447446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.447419    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:28.448282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.448261    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo" pod "3650035e-9a85-4eee-8dd4-a2bab7815442" (UID: "3650035e-9a85-4eee-8dd4-a2bab7815442"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:36:28.550031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.549986    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bb98\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-kube-api-access-9bb98\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.550031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.550028    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.550234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.550042    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.550234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.550054    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3650035e-9a85-4eee-8dd4-a2bab7815442-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.554209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.554186    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:28.651056 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.651019    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb\" (UniqueName: \"kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb\") pod \"6be8ca25-8b9e-4166-9599-798196ac0bc7\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") "
Oct 31 16:36:28.651260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.651123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca\") pod \"6be8ca25-8b9e-4166-9599-798196ac0bc7\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") "
Oct 31 16:36:28.651521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.651485    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca\") pod \"6be8ca25-8b9e-4166-9599-798196ac0bc7\" (UID: \"6be8ca25-8b9e-4166-9599-798196ac0bc7\") "
Oct 31 16:36:28.652311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.652282    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca" pod "6be8ca25-8b9e-4166-9599-798196ac0bc7" (UID: "6be8ca25-8b9e-4166-9599-798196ac0bc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:28.663166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.653684    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb" pod "6be8ca25-8b9e-4166-9599-798196ac0bc7" (UID: "6be8ca25-8b9e-4166-9599-798196ac0bc7"). InnerVolumeSpecName "kube-api-access-kxmsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:28.663166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.653771    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca" pod "6be8ca25-8b9e-4166-9599-798196ac0bc7" (UID: "6be8ca25-8b9e-4166-9599-798196ac0bc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:28.754175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.754145    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.754175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.754174    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmsb\" (UniqueName: \"kubernetes.io/projected/6be8ca25-8b9e-4166-9599-798196ac0bc7-kube-api-access-kxmsb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:28.754390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:28.754184    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be8ca25-8b9e-4166-9599-798196ac0bc7-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:29.425509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:29.425422    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"
Oct 31 16:36:29.426944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:29.426921    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"
Oct 31 16:36:29.427102 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:29.426916    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk" event={"ID":"6be8ca25-8b9e-4166-9599-798196ac0bc7","Type":"ContainerDied","Data":"00f79ed30466630c7677e9d459a50bf5fc13abb36b692a1f59cd128ff849c296"}
Oct 31 16:36:29.427102 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:29.427051    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00f79ed30466630c7677e9d459a50bf5fc13abb36b692a1f59cd128ff849c296"
Oct 31 16:36:29.692734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:29.692659    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"]
Oct 31 16:36:29.696660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:29.696626    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ae37-openshift-default-67d57d5d77-qjgr4"]
Oct 31 16:36:30.440464 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:30.440433    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" path="/var/lib/kubelet/pods/3650035e-9a85-4eee-8dd4-a2bab7815442/volumes"
Oct 31 16:36:32.908807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:32.908769    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"]
Oct 31 16:36:32.954442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:32.954395    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6b1cca-g8vw7"]
Oct 31 16:36:34.226167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:34.226132    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"]
Oct 31 16:36:34.279509 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:34.275542    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-822dd4-r6x7s"]
Oct 31 16:36:34.454790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:34.454757    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c9c006-67c5-4e45-b613-13cb7e63e152" path="/var/lib/kubelet/pods/53c9c006-67c5-4e45-b613-13cb7e63e152/volumes"
Oct 31 16:36:34.455650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:34.455628    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9465907-2772-48ca-beb0-e598129ee1ce" path="/var/lib/kubelet/pods/d9465907-2772-48ca-beb0-e598129ee1ce/volumes"
Oct 31 16:36:35.658769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:35.658730    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"]
Oct 31 16:36:35.659186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:35.659009    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy" containerID="cri-o://d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4" gracePeriod=30
Oct 31 16:36:36.768643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:36.768594    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"]
Oct 31 16:36:40.215432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.215395    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"]
Oct 31 16:36:40.216103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216089    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6be8ca25-8b9e-4166-9599-798196ac0bc7" containerName="revoke-keys"
Oct 31 16:36:40.216149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216104    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be8ca25-8b9e-4166-9599-798196ac0bc7" containerName="revoke-keys"
Oct 31 16:36:40.216149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216132    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy"
Oct 31 16:36:40.216149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216137    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy"
Oct 31 16:36:40.216149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216146    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy"
Oct 31 16:36:40.216283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216151    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy"
Oct 31 16:36:40.216283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216258    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be8ca25-8b9e-4166-9599-798196ac0bc7" containerName="revoke-keys"
Oct 31 16:36:40.216283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216270    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2128ce4c-cc8b-4f0e-af6c-d3f4db820f1a" containerName="istio-proxy"
Oct 31 16:36:40.216283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216279    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy"
Oct 31 16:36:40.216411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.216286    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3650035e-9a85-4eee-8dd4-a2bab7815442" containerName="istio-proxy"
Oct 31 16:36:40.220964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.220943    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.272810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.272778    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"]
Oct 31 16:36:40.314713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.314683    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:36:40.314713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.314683    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:36:40.314895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.314731    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:36:40.317386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.317364    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:36:40.357211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.357176    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7t5p\" (UniqueName: \"kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.357371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.357235    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.357496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.357446    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.458930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.458898    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7t5p\" (UniqueName: \"kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.459104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.458948    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.459104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.459074    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.459753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.459728    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.461339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.461309    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.512776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.512705    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7t5p\" (UniqueName: \"kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.535218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.535189    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:40.747303 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:36:40.747271    4519 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861b064c_4ced_4226_aaee_915a2c9c4019.slice/crio-d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:36:40.910132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.910102    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"]
Oct 31 16:36:40.911054 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:36:40.911027    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2585c09f_a872_4c4d_9cd5_2c444600083d.slice/crio-9a325f28eb36a846e4f90b3246a97bd70d554a47148cd5c98d1749ec44c4e1ba WatchSource:0}: Error finding container 9a325f28eb36a846e4f90b3246a97bd70d554a47148cd5c98d1749ec44c4e1ba: Status 404 returned error can't find the container with id 9a325f28eb36a846e4f90b3246a97bd70d554a47148cd5c98d1749ec44c4e1ba
Oct 31 16:36:40.923864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:40.923839    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085113    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085171    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085232    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085389    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085448    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085507    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085550    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085600    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.085665    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo\") pod \"861b064c-4ced-4226-aaee-915a2c9c4019\" (UID: \"861b064c-4ced-4226-aaee-915a2c9c4019\") "
Oct 31 16:36:41.088545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.087793    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:41.089242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.088649    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:41.089242 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.088883    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:41.089360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.089321    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:41.114905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.089493    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:41.114905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.098478    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b064c-4ced-4226-aaee-915a2c9c4019-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.114905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.098507    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.114905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.098525    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.114905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.098542    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.114905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.098558    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.115334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.114929    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:36:41.117690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.117641    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:41.117959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.117925    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "kube-api-access-28q8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:41.187836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.187787    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token" pod "861b064c-4ced-4226-aaee-915a2c9c4019" (UID: "861b064c-4ced-4226-aaee-915a2c9c4019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:41.201733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.201694    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28q8j\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-kube-api-access-28q8j\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.201979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.201963    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b064c-4ced-4226-aaee-915a2c9c4019-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.202113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.202100    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b064c-4ced-4226-aaee-915a2c9c4019-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.202213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.202201    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b064c-4ced-4226-aaee-915a2c9c4019-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:41.553771 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.553740    4519 generic.go:356] "Generic (PLEG): container finished" podID="861b064c-4ced-4226-aaee-915a2c9c4019" containerID="d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4" exitCode=0
Oct 31 16:36:41.554207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.553789    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"
Oct 31 16:36:41.554207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.553823    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" event={"ID":"861b064c-4ced-4226-aaee-915a2c9c4019","Type":"ContainerDied","Data":"d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4"}
Oct 31 16:36:41.554207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.553876    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2" event={"ID":"861b064c-4ced-4226-aaee-915a2c9c4019","Type":"ContainerDied","Data":"1c2d2677f0198b041e1c0bd4dd7ec0ade2cefc23ee8fab0d884170f4bc5ebd82"}
Oct 31 16:36:41.554207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.553899    4519 scope.go:122] "RemoveContainer" containerID="d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4"
Oct 31 16:36:41.555975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.555953    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r" event={"ID":"2585c09f-a872-4c4d-9cd5-2c444600083d","Type":"ContainerStarted","Data":"c108a4f1516b6039ed2e3fac3d192038bef9a8f9eb30f506f3aff72107ad3650"}
Oct 31 16:36:41.556115 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.555991    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r" event={"ID":"2585c09f-a872-4c4d-9cd5-2c444600083d","Type":"ContainerStarted","Data":"9a325f28eb36a846e4f90b3246a97bd70d554a47148cd5c98d1749ec44c4e1ba"}
Oct 31 16:36:41.565156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.565136    4519 scope.go:122] "RemoveContainer" containerID="d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4"
Oct 31 16:36:41.565416 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:36:41.565399    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4\": container with ID starting with d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4 not found: ID does not exist" containerID="d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4"
Oct 31 16:36:41.565486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.565426    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4"} err="failed to get container status \"d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4\": rpc error: code = NotFound desc = could not find container \"d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4\": container with ID starting with d3d3a140ca4545ed3a436f71c3e9e019c5f95145928b216888a4f3f3dfa91ac4 not found: ID does not exist"
Oct 31 16:36:41.716173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.716121    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r" podStartSLOduration=1.71610847 podStartE2EDuration="1.71610847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:41.713782549 +0000 UTC m=+2045.413711859" watchObservedRunningTime="2026-10-31 16:36:41.71610847 +0000 UTC m=+2045.416037780"
Oct 31 16:36:41.733342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.733274    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"]
Oct 31 16:36:41.734178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.734149    4519 cpu_manager.go:414] "containerMap: removing container" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy"
Oct 31 16:36:41.734178 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.734172    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy"
Oct 31 16:36:41.734397 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.734343    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" containerName="istio-proxy"
Oct 31 16:36:41.739367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.739346    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:41.769084 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.769054    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl" podUID="52bdbf93-f01c-4801-8fec-71e606450eaa" containerName="payload-processing" containerID="cri-o://6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e" gracePeriod=25
Oct 31 16:36:41.788984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.788954    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"]
Oct 31 16:36:41.826314 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.826278    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"]
Oct 31 16:36:41.832303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.832269    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-lwfw2"]
Oct 31 16:36:41.913818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.913270    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:41.913818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.913500    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25cp\" (UniqueName: \"kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:41.913818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:41.913635    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.015063 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.014959    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25cp\" (UniqueName: \"kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.015524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.015506    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.015876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.015841    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.016967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.016943    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.019350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.019314    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.059883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.059854    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:36:42.069395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.069370    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25cp\" (UniqueName: \"kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.217760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.217723    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume\") pod \"52bdbf93-f01c-4801-8fec-71e606450eaa\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") "
Oct 31 16:36:42.217933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.217877    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668\" (UniqueName: \"kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668\") pod \"52bdbf93-f01c-4801-8fec-71e606450eaa\" (UID: \"52bdbf93-f01c-4801-8fec-71e606450eaa\") "
Oct 31 16:36:42.218185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.218157    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume" pod "52bdbf93-f01c-4801-8fec-71e606450eaa" (UID: "52bdbf93-f01c-4801-8fec-71e606450eaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:42.218603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.218584    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bdbf93-f01c-4801-8fec-71e606450eaa-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:42.219933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.219915    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668" pod "52bdbf93-f01c-4801-8fec-71e606450eaa" (UID: "52bdbf93-f01c-4801-8fec-71e606450eaa"). InnerVolumeSpecName "kube-api-access-qv668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:42.320350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.320269    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv668\" (UniqueName: \"kubernetes.io/projected/52bdbf93-f01c-4801-8fec-71e606450eaa-kube-api-access-qv668\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:42.352979 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.352945    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:42.444060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.444027    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861b064c-4ced-4226-aaee-915a2c9c4019" path="/var/lib/kubelet/pods/861b064c-4ced-4226-aaee-915a2c9c4019/volumes"
Oct 31 16:36:42.564325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.564292    4519 generic.go:356] "Generic (PLEG): container finished" podID="52bdbf93-f01c-4801-8fec-71e606450eaa" containerID="6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e" exitCode=0
Oct 31 16:36:42.564828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.564350    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"
Oct 31 16:36:42.564828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.564373    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl" event={"ID":"52bdbf93-f01c-4801-8fec-71e606450eaa","Type":"ContainerDied","Data":"6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e"}
Oct 31 16:36:42.564828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.564419    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl" event={"ID":"52bdbf93-f01c-4801-8fec-71e606450eaa","Type":"ContainerDied","Data":"a7941d4db693814afed7d90d8b2c9e673c8c5f1325df39060905a45cbf176edb"}
Oct 31 16:36:42.564828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.564480    4519 scope.go:122] "RemoveContainer" containerID="6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e"
Oct 31 16:36:42.566419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.566395    4519 generic.go:356] "Generic (PLEG): container finished" podID="2585c09f-a872-4c4d-9cd5-2c444600083d" containerID="c108a4f1516b6039ed2e3fac3d192038bef9a8f9eb30f506f3aff72107ad3650" exitCode=0
Oct 31 16:36:42.566574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.566494    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r" event={"ID":"2585c09f-a872-4c4d-9cd5-2c444600083d","Type":"ContainerDied","Data":"c108a4f1516b6039ed2e3fac3d192038bef9a8f9eb30f506f3aff72107ad3650"}
Oct 31 16:36:42.574939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.574913    4519 scope.go:122] "RemoveContainer" containerID="6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e"
Oct 31 16:36:42.575211 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:36:42.575186    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e\": container with ID starting with 6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e not found: ID does not exist" containerID="6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e"
Oct 31 16:36:42.575308 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.575219    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e"} err="failed to get container status \"6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e\": rpc error: code = NotFound desc = could not find container \"6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e\": container with ID starting with 6b34c71e5b0a7299bca1cc342dc68b1f08375d0e7e66ec0f2784e8b190b5825e not found: ID does not exist"
Oct 31 16:36:42.723980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.723849    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"]
Oct 31 16:36:42.731322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.730233    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"]
Oct 31 16:36:42.735031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:42.735005    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc67dd-5859967d4f-p2prl"]
Oct 31 16:36:43.591346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.591299    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth" event={"ID":"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d","Type":"ContainerStarted","Data":"82144edd2245b95f218db777fbf763d49cba662753a6cb28b7d3dc7649827624"}
Oct 31 16:36:43.591839 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.591397    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth" event={"ID":"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d","Type":"ContainerStarted","Data":"2bc8f3a2e6879e34c4aa9a3a277484a9ad2c1f3406e63d8715e552a130e55d7a"}
Oct 31 16:36:43.700263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.700201    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth" podStartSLOduration=2.700179207 podStartE2EDuration="2.700179207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:43.698535099 +0000 UTC m=+2047.398464437" watchObservedRunningTime="2026-10-31 16:36:43.700179207 +0000 UTC m=+2047.400108520"
Oct 31 16:36:43.781984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.781958    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:43.948992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.948909    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca\") pod \"2585c09f-a872-4c4d-9cd5-2c444600083d\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") "
Oct 31 16:36:43.949160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.949076    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p\" (UniqueName: \"kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p\") pod \"2585c09f-a872-4c4d-9cd5-2c444600083d\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") "
Oct 31 16:36:43.949245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.949155    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca\") pod \"2585c09f-a872-4c4d-9cd5-2c444600083d\" (UID: \"2585c09f-a872-4c4d-9cd5-2c444600083d\") "
Oct 31 16:36:43.949334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.949309    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca" pod "2585c09f-a872-4c4d-9cd5-2c444600083d" (UID: "2585c09f-a872-4c4d-9cd5-2c444600083d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:43.949847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.949823    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:43.951172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.951149    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca" pod "2585c09f-a872-4c4d-9cd5-2c444600083d" (UID: "2585c09f-a872-4c4d-9cd5-2c444600083d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:43.951263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:43.951228    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p" pod "2585c09f-a872-4c4d-9cd5-2c444600083d" (UID: "2585c09f-a872-4c4d-9cd5-2c444600083d"). InnerVolumeSpecName "kube-api-access-v7t5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:44.051004 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.050968    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7t5p\" (UniqueName: \"kubernetes.io/projected/2585c09f-a872-4c4d-9cd5-2c444600083d-kube-api-access-v7t5p\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.051004 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.050998    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2585c09f-a872-4c4d-9cd5-2c444600083d-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.440645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.440613    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bdbf93-f01c-4801-8fec-71e606450eaa" path="/var/lib/kubelet/pods/52bdbf93-f01c-4801-8fec-71e606450eaa/volumes"
Oct 31 16:36:44.601615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.601584    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"
Oct 31 16:36:44.602099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.601622    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r" event={"ID":"2585c09f-a872-4c4d-9cd5-2c444600083d","Type":"ContainerDied","Data":"9a325f28eb36a846e4f90b3246a97bd70d554a47148cd5c98d1749ec44c4e1ba"}
Oct 31 16:36:44.602099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.601673    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a325f28eb36a846e4f90b3246a97bd70d554a47148cd5c98d1749ec44c4e1ba"
Oct 31 16:36:44.604250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.604224    4519 generic.go:356] "Generic (PLEG): container finished" podID="8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" containerID="82144edd2245b95f218db777fbf763d49cba662753a6cb28b7d3dc7649827624" exitCode=0
Oct 31 16:36:44.604389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:44.604300    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth" event={"ID":"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d","Type":"ContainerDied","Data":"82144edd2245b95f218db777fbf763d49cba662753a6cb28b7d3dc7649827624"}
Oct 31 16:36:45.782240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.782214    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:45.877970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.877890    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp\" (UniqueName: \"kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp\") pod \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") "
Oct 31 16:36:45.878140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.878013    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca\") pod \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") "
Oct 31 16:36:45.878203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.878185    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca\") pod \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\" (UID: \"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d\") "
Oct 31 16:36:45.878553 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.878526    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca" pod "8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" (UID: "8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:45.878764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.878744    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:45.880106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.880084    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca" pod "8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" (UID: "8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:45.880203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.880118    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp" pod "8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" (UID: "8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d"). InnerVolumeSpecName "kube-api-access-b25cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:45.980365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.980329    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25cp\" (UniqueName: \"kubernetes.io/projected/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-kube-api-access-b25cp\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:45.980365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:45.980357    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.622872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:46.622841    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"
Oct 31 16:36:46.623081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:46.622847    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth" event={"ID":"8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d","Type":"ContainerDied","Data":"2bc8f3a2e6879e34c4aa9a3a277484a9ad2c1f3406e63d8715e552a130e55d7a"}
Oct 31 16:36:46.623081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:46.622969    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc8f3a2e6879e34c4aa9a3a277484a9ad2c1f3406e63d8715e552a130e55d7a"
Oct 31 16:36:54.005748 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:36:54.005701    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:36:58.446483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:36:58.446429    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:37:16.266036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:16.265982    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"]
Oct 31 16:37:16.313226 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:16.313196    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc67dd-8lpkk"]
Oct 31 16:37:16.440796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:16.440762    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be8ca25-8b9e-4166-9599-798196ac0bc7" path="/var/lib/kubelet/pods/6be8ca25-8b9e-4166-9599-798196ac0bc7/volumes"
Oct 31 16:37:22.833150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:22.833112    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"]
Oct 31 16:37:22.834095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:22.833406    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" containerID="cri-o://985ce94d616582c21d6c88379c8e3f82a06c0c2106ad6cdabdd48944e6f66f78" gracePeriod=30
Oct 31 16:37:23.701002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.700967    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"]
Oct 31 16:37:23.701673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701656    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" containerName="revoke-keys"
Oct 31 16:37:23.701673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701673    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" containerName="revoke-keys"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701685    4519 cpu_manager.go:414] "containerMap: removing container" podUID="52bdbf93-f01c-4801-8fec-71e606450eaa" containerName="payload-processing"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701691    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bdbf93-f01c-4801-8fec-71e606450eaa" containerName="payload-processing"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701706    4519 cpu_manager.go:414] "containerMap: removing container" podUID="2585c09f-a872-4c4d-9cd5-2c444600083d" containerName="revoke-keys"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701711    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2585c09f-a872-4c4d-9cd5-2c444600083d" containerName="revoke-keys"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701831    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bdbf93-f01c-4801-8fec-71e606450eaa" containerName="payload-processing"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701842    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="2585c09f-a872-4c4d-9cd5-2c444600083d" containerName="revoke-keys"
Oct 31 16:37:23.701855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.701855    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" containerName="revoke-keys"
Oct 31 16:37:23.705855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.705836    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.747046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.747017    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.747169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.747140    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8hh\" (UniqueName: \"kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.747217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.747172    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.758673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.758643    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"]
Oct 31 16:37:23.800961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.800927    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:37:23.801141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.801125    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:37:23.801227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.801211    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:37:23.802539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.802522    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:37:23.848737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.848712    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.849155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.848799    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8hh\" (UniqueName: \"kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.849155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.848846    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.849582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.849563    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.851112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.851088    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:23.858166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.858120    4519 patch_prober.go:29] interesting pod/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:37:23.858286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.858209    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:37:23.902583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:23.902562    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8hh\" (UniqueName: \"kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:24.004638 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:37:24.004595    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:37:24.019487 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:24.019428    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:24.392913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:24.392888    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"]
Oct 31 16:37:24.392913 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:37:24.392906    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccf1b4a_4e0b_47dd_a08e_3241f00d6154.slice/crio-7ff3f67c067d8fe8ad20ca53020b7eee0d77ab6ae5749de93ef93bcc502d3e6d WatchSource:0}: Error finding container 7ff3f67c067d8fe8ad20ca53020b7eee0d77ab6ae5749de93ef93bcc502d3e6d: Status 404 returned error can't find the container with id 7ff3f67c067d8fe8ad20ca53020b7eee0d77ab6ae5749de93ef93bcc502d3e6d
Oct 31 16:37:25.024218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.024116    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk" event={"ID":"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154","Type":"ContainerStarted","Data":"aeda2b7312afcf1c9705d9455a1188ee85ba0f3c3b5fc3503e3add2a3fc58fe1"}
Oct 31 16:37:25.024218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.024182    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk" event={"ID":"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154","Type":"ContainerStarted","Data":"7ff3f67c067d8fe8ad20ca53020b7eee0d77ab6ae5749de93ef93bcc502d3e6d"}
Oct 31 16:37:25.135323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.135266    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk" podStartSLOduration=2.135251283 podStartE2EDuration="2.135251283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:25.130865147 +0000 UTC m=+2088.830794459" watchObservedRunningTime="2026-10-31 16:37:25.135251283 +0000 UTC m=+2088.835180593"
Oct 31 16:37:25.513405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.513372    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"]
Oct 31 16:37:25.513610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.513585    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy" containerID="cri-o://7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d" gracePeriod=30
Oct 31 16:37:25.568339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.568308    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"]
Oct 31 16:37:25.568591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.568556    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy" containerID="cri-o://91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462" gracePeriod=30
Oct 31 16:37:25.713587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.713558    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"]
Oct 31 16:37:25.713795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:25.713754    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" containerID="cri-o://82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859" gracePeriod=30
Oct 31 16:37:26.034603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:26.034510    4519 generic.go:356] "Generic (PLEG): container finished" podID="3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" containerID="aeda2b7312afcf1c9705d9455a1188ee85ba0f3c3b5fc3503e3add2a3fc58fe1" exitCode=0
Oct 31 16:37:26.034980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:26.034593    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk" event={"ID":"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154","Type":"ContainerDied","Data":"aeda2b7312afcf1c9705d9455a1188ee85ba0f3c3b5fc3503e3add2a3fc58fe1"}
Oct 31 16:37:27.212192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.212161    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:27.302975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.302890    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca\") pod \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") "
Oct 31 16:37:27.303129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.303012    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh\" (UniqueName: \"kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh\") pod \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") "
Oct 31 16:37:27.303187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.303144    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca\") pod \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\" (UID: \"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154\") "
Oct 31 16:37:27.303571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.303532    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca" pod "3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" (UID: "3ccf1b4a-4e0b-47dd-a08e-3241f00d6154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:27.303808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.303790    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:27.305040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.305004    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca" pod "3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" (UID: "3ccf1b4a-4e0b-47dd-a08e-3241f00d6154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:37:27.305275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.305252    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh" pod "3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" (UID: "3ccf1b4a-4e0b-47dd-a08e-3241f00d6154"). InnerVolumeSpecName "kube-api-access-gs8hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:27.405401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.405364    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:27.405401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:27.405398    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs8hh\" (UniqueName: \"kubernetes.io/projected/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154-kube-api-access-gs8hh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.073446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.072556    4519 generic.go:356] "Generic (PLEG): container finished" podID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerID="985ce94d616582c21d6c88379c8e3f82a06c0c2106ad6cdabdd48944e6f66f78" exitCode=0
Oct 31 16:37:28.073446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.072606    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" event={"ID":"40d9cc78-f6fb-4031-a540-adc0d2e4fadd","Type":"ContainerDied","Data":"985ce94d616582c21d6c88379c8e3f82a06c0c2106ad6cdabdd48944e6f66f78"}
Oct 31 16:37:28.073446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.073021    4519 scope.go:122] "RemoveContainer" containerID="76b24565c90e9a45bbf3017326f917216b3f7a4a5f6b507013e55d3053b298cc"
Oct 31 16:37:28.078238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.078201    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk" event={"ID":"3ccf1b4a-4e0b-47dd-a08e-3241f00d6154","Type":"ContainerDied","Data":"7ff3f67c067d8fe8ad20ca53020b7eee0d77ab6ae5749de93ef93bcc502d3e6d"}
Oct 31 16:37:28.078365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.078268    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff3f67c067d8fe8ad20ca53020b7eee0d77ab6ae5749de93ef93bcc502d3e6d"
Oct 31 16:37:28.078434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.078365    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"
Oct 31 16:37:28.114279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.114260    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:37:28.215667 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.215641    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.215689    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.215712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.215785    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.215862    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.215918    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216076    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:28.216301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216146    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216157    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:28.216301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216181    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216225    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket\") pod \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\" (UID: \"40d9cc78-f6fb-4031-a540-adc0d2e4fadd\") "
Oct 31 16:37:28.216301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216249    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:28.216690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216545    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:28.216690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216607    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:28.216870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216852    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.216938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216872    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.216938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216882    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.216938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216891    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.216938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.216899    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.218227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.218203    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:28.218319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.218235    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:28.218415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.218398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:28.218484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.218420    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt" pod "40d9cc78-f6fb-4031-a540-adc0d2e4fadd" (UID: "40d9cc78-f6fb-4031-a540-adc0d2e4fadd"). InnerVolumeSpecName "kube-api-access-vgrlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:28.318696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.318658    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.318696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.318690    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.318696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.318699    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:28.318939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:28.318709    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgrlt\" (UniqueName: \"kubernetes.io/projected/40d9cc78-f6fb-4031-a540-adc0d2e4fadd-kube-api-access-vgrlt\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:29.087336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:29.087305    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"
Oct 31 16:37:29.087610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:29.087341    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz" event={"ID":"40d9cc78-f6fb-4031-a540-adc0d2e4fadd","Type":"ContainerDied","Data":"97ea85423e427a349a512fa5bf991c3648c95e5af67e08d9d3a3e3c7ebf05898"}
Oct 31 16:37:29.087610 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:29.087405    4519 scope.go:122] "RemoveContainer" containerID="985ce94d616582c21d6c88379c8e3f82a06c0c2106ad6cdabdd48944e6f66f78"
Oct 31 16:37:29.255952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:29.255924    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"]
Oct 31 16:37:29.302895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:29.302865    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dc67dd-openshift-default-587b579d8c-d98fz"]
Oct 31 16:37:30.409952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.409918    4519 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:37:30.410384 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.409969    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:37:30.440997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.440967    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" path="/var/lib/kubelet/pods/40d9cc78-f6fb-4031-a540-adc0d2e4fadd/volumes"
Oct 31 16:37:30.878281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.878260    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:37:30.883695 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.883668    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:37:30.961967 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.961941    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.961975    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962019    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962066    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962185    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962228    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962298    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962355    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962446    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:30.962494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962477    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b\") pod \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\" (UID: \"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9\") "
Oct 31 16:37:30.962660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962531    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:30.962817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962759    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:30.962883 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962833    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:30.963182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.962956    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:30.963276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.963187    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:30.963276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.963203    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:30.963276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.963213    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:30.963276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.963227    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:30.963276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.963238    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:30.964982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.964954    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "kube-api-access-59h5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:30.965541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.965515    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:30.965641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.965554    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:30.965641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.965636    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo" pod "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" (UID: "e4c0badf-2977-4f9a-8b8b-37d295c3f1a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:30.998993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:30.998965    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:37:31.064073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064039    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064083    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064151    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064209    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064317    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064361    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.064536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064368    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064448    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.064747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064637    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.064809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064752    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.064871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064831    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:31.065009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.064983    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket\") pod \"d8f04b95-169f-4b49-8af6-f48c677b6def\" (UID: \"d8f04b95-169f-4b49-8af6-f48c677b6def\") "
Oct 31 16:37:31.065590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065428    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.065859 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065841    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.065947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065865    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.065947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065880    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.065947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065893    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.065947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065906    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f04b95-169f-4b49-8af6-f48c677b6def-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.065947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065921    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59h5b\" (UniqueName: \"kubernetes.io/projected/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-kube-api-access-59h5b\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.065947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065935    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.066160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065949    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.066160 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.065963    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.067217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.067192    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.067315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.067287    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "kube-api-access-h75mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:31.067373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.067357    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:31.067411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.067381    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token" pod "d8f04b95-169f-4b49-8af6-f48c677b6def" (UID: "d8f04b95-169f-4b49-8af6-f48c677b6def"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:31.115808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.115766    4519 generic.go:356] "Generic (PLEG): container finished" podID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerID="91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462" exitCode=0
Oct 31 16:37:31.115997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.115829    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"
Oct 31 16:37:31.115997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.115891    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" event={"ID":"d8f04b95-169f-4b49-8af6-f48c677b6def","Type":"ContainerDied","Data":"91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462"}
Oct 31 16:37:31.115997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.115955    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8" event={"ID":"d8f04b95-169f-4b49-8af6-f48c677b6def","Type":"ContainerDied","Data":"604a670f1ae9842e5531951e3d3331e5ae956b59636e5a189da67d3139190ae1"}
Oct 31 16:37:31.115997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.115976    4519 scope.go:122] "RemoveContainer" containerID="91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462"
Oct 31 16:37:31.119947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.119917    4519 generic.go:356] "Generic (PLEG): container finished" podID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerID="7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d" exitCode=0
Oct 31 16:37:31.120164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.120138    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"
Oct 31 16:37:31.120164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.120148    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" event={"ID":"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9","Type":"ContainerDied","Data":"7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d"}
Oct 31 16:37:31.120164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.120192    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp" event={"ID":"e4c0badf-2977-4f9a-8b8b-37d295c3f1a9","Type":"ContainerDied","Data":"98562429c75f007b5b9118638da9d1d85af905b5dcb82fb1a83f7bf5c824fe2e"}
Oct 31 16:37:31.121980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.121848    4519 generic.go:356] "Generic (PLEG): container finished" podID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerID="82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859" exitCode=0
Oct 31 16:37:31.122152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.122012    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" event={"ID":"aa01192e-c28e-43c6-8780-7afcb8be04a7","Type":"ContainerDied","Data":"82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859"}
Oct 31 16:37:31.122152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.122060    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5" event={"ID":"aa01192e-c28e-43c6-8780-7afcb8be04a7","Type":"ContainerDied","Data":"e079c4994759f38b991aec7cee5003996ef9562cb4e8db068c7a3dd93d8b01bf"}
Oct 31 16:37:31.122152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.122133    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"
Oct 31 16:37:31.131946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.131913    4519 scope.go:122] "RemoveContainer" containerID="91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462"
Oct 31 16:37:31.132288 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:37:31.132248    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462\": container with ID starting with 91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462 not found: ID does not exist" containerID="91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462"
Oct 31 16:37:31.132729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.132696    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462"} err="failed to get container status \"91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462\": rpc error: code = NotFound desc = could not find container \"91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462\": container with ID starting with 91ace5cc75eaa44d45b54acdb87cb9dae0e62c1ba9bfa3eec4a46d45d26be462 not found: ID does not exist"
Oct 31 16:37:31.132843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.132731    4519 scope.go:122] "RemoveContainer" containerID="7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d"
Oct 31 16:37:31.144289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.144268    4519 scope.go:122] "RemoveContainer" containerID="7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d"
Oct 31 16:37:31.146667 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:37:31.146641    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d\": container with ID starting with 7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d not found: ID does not exist" containerID="7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d"
Oct 31 16:37:31.146741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.146680    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d"} err="failed to get container status \"7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d\": rpc error: code = NotFound desc = could not find container \"7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d\": container with ID starting with 7e093947edc35d4dd58a8d019669e547c28db07ae76e2c9e5bd15fc2143f347d not found: ID does not exist"
Oct 31 16:37:31.146741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.146704    4519 scope.go:122] "RemoveContainer" containerID="82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859"
Oct 31 16:37:31.159439 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.159420    4519 scope.go:122] "RemoveContainer" containerID="82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859"
Oct 31 16:37:31.159770 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:37:31.159744    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859\": container with ID starting with 82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859 not found: ID does not exist" containerID="82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859"
Oct 31 16:37:31.159861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.159782    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859"} err="failed to get container status \"82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859\": rpc error: code = NotFound desc = could not find container \"82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859\": container with ID starting with 82d3f270943b0ecb874ec6dfcaa18dc5112f4730bbefb0bee688b746bf951859 not found: ID does not exist"
Oct 31 16:37:31.166374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.166355    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.166506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.166392    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.166506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.166420    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.166680 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.166661    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.166845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.166818    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:31.166989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.166950    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.167064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167050    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.167129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167088    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.167249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167228    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.167320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167268    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.167378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167333    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data\") pod \"aa01192e-c28e-43c6-8780-7afcb8be04a7\" (UID: \"aa01192e-c28e-43c6-8780-7afcb8be04a7\") "
Oct 31 16:37:31.167436 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167420    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.167511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167491    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.167710    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168081    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168104    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168122    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168136    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168151    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168166    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h75mn\" (UniqueName: \"kubernetes.io/projected/d8f04b95-169f-4b49-8af6-f48c677b6def-kube-api-access-h75mn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168179    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168193    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa01192e-c28e-43c6-8780-7afcb8be04a7-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.168229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.168207    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f04b95-169f-4b49-8af6-f48c677b6def-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.169151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.169126    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:31.169614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.169572    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "kube-api-access-5v7d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:31.169807 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.169784    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:31.169966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.169949    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy" pod "aa01192e-c28e-43c6-8780-7afcb8be04a7" (UID: "aa01192e-c28e-43c6-8780-7afcb8be04a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:31.281505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.281390    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.281505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.281431    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v7d2\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-kube-api-access-5v7d2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.281505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.281447    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.281505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.281495    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa01192e-c28e-43c6-8780-7afcb8be04a7-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:31.379666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.379631    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"]
Oct 31 16:37:31.384032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.384000    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-vggc8"]
Oct 31 16:37:31.485337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.485297    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"]
Oct 31 16:37:31.491225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.491199    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-rg6fp"]
Oct 31 16:37:31.606475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.601278    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"]
Oct 31 16:37:31.606475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:31.606238    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-t95p5"]
Oct 31 16:37:32.440602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:32.440567    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" path="/var/lib/kubelet/pods/aa01192e-c28e-43c6-8780-7afcb8be04a7/volumes"
Oct 31 16:37:32.441195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:32.441173    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" path="/var/lib/kubelet/pods/d8f04b95-169f-4b49-8af6-f48c677b6def/volumes"
Oct 31 16:37:32.441798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:32.441774    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" path="/var/lib/kubelet/pods/e4c0badf-2977-4f9a-8b8b-37d295c3f1a9/volumes"
Oct 31 16:37:42.847447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:42.847412    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"]
Oct 31 16:37:42.848002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:42.847667    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" containerID="cri-o://023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3" gracePeriod=30
Oct 31 16:37:46.508801 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:46.508766    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tc5qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:37:46.509189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:46.508831    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:37:48.137190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.137163    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:37:48.201093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201060    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201168    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201208    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201234    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201313    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201346    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201380    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201404    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:48.201747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201519    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201592    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh\") pod \"90823085-6810-4c40-a994-1f59f903032f\" (UID: \"90823085-6810-4c40-a994-1f59f903032f\") "
Oct 31 16:37:48.201747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201681    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:48.201747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201692    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:48.202053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201740    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:48.202053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.201921    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:48.202363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.202329    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.202363 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.202356    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.202544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.202373    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90823085-6810-4c40-a994-1f59f903032f-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.202544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.202390    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.202544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.202405    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.203777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.203739    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:48.203902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.203857    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:48.203994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.203969    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "kube-api-access-r2rrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:48.204042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.204025    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy" pod "90823085-6810-4c40-a994-1f59f903032f" (UID: "90823085-6810-4c40-a994-1f59f903032f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:48.255180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.255143    4519 generic.go:356] "Generic (PLEG): container finished" podID="90823085-6810-4c40-a994-1f59f903032f" containerID="023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3" exitCode=0
Oct 31 16:37:48.255349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.255198    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"
Oct 31 16:37:48.255349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.255225    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" event={"ID":"90823085-6810-4c40-a994-1f59f903032f","Type":"ContainerDied","Data":"023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3"}
Oct 31 16:37:48.255349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.255313    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb" event={"ID":"90823085-6810-4c40-a994-1f59f903032f","Type":"ContainerDied","Data":"9cf9002ad83fff7220c401a76c871feea2c6d1090bc3bc8ebc7c93e2b3800963"}
Oct 31 16:37:48.255349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.255338    4519 scope.go:122] "RemoveContainer" containerID="023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3"
Oct 31 16:37:48.267292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.267272    4519 scope.go:122] "RemoveContainer" containerID="023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3"
Oct 31 16:37:48.267584 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:37:48.267564    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3\": container with ID starting with 023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3 not found: ID does not exist" containerID="023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3"
Oct 31 16:37:48.267666 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.267595    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3"} err="failed to get container status \"023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3\": rpc error: code = NotFound desc = could not find container \"023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3\": container with ID starting with 023a68a67890817f496c30a38e13542b6dc233ca1cd4dcfa43bfcfb8a4d14bc3 not found: ID does not exist"
Oct 31 16:37:48.303573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.303545    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90823085-6810-4c40-a994-1f59f903032f-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.303573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.303571    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.303764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.303582    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90823085-6810-4c40-a994-1f59f903032f-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.303764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.303592    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2rrh\" (UniqueName: \"kubernetes.io/projected/90823085-6810-4c40-a994-1f59f903032f-kube-api-access-r2rrh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:48.405377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.405297    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"]
Oct 31 16:37:48.409658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.409628    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tc5qb"]
Oct 31 16:37:48.439877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:37:48.439845    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90823085-6810-4c40-a994-1f59f903032f" path="/var/lib/kubelet/pods/90823085-6810-4c40-a994-1f59f903032f/volumes"
Oct 31 16:37:54.009198 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:37:54.009160    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:38:01.891476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:01.886051    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"]
Oct 31 16:38:01.953515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:01.953481    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dac4fa-59phz"]
Oct 31 16:38:02.440850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:02.440816    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c977453d-86d9-4d01-bcda-f5996605e517" path="/var/lib/kubelet/pods/c977453d-86d9-4d01-bcda-f5996605e517/volumes"
Oct 31 16:38:03.714271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:03.714238    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"]
Oct 31 16:38:03.761170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:03.761134    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-487213-76f5r"]
Oct 31 16:38:04.440170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:04.440140    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2585c09f-a872-4c4d-9cd5-2c444600083d" path="/var/lib/kubelet/pods/2585c09f-a872-4c4d-9cd5-2c444600083d/volumes"
Oct 31 16:38:06.342173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342138    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"]
Oct 31 16:38:06.342867 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342850    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy"
Oct 31 16:38:06.342914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342868    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy"
Oct 31 16:38:06.342914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342886    4519 cpu_manager.go:414] "containerMap: removing container" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy"
Oct 31 16:38:06.342914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342891    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy"
Oct 31 16:38:06.342914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342902    4519 cpu_manager.go:414] "containerMap: removing container" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy"
Oct 31 16:38:06.342914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342906    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342919    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342925    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342939    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342944    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342960    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" containerName="revoke-keys"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342965    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" containerName="revoke-keys"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342977    4519 cpu_manager.go:414] "containerMap: removing container" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy"
Oct 31 16:38:06.343078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.342981    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343081    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c0badf-2977-4f9a-8b8b-37d295c3f1a9" containerName="istio-proxy"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343089    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa01192e-c28e-43c6-8780-7afcb8be04a7" containerName="istio-proxy"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343098    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="90823085-6810-4c40-a994-1f59f903032f" containerName="istio-proxy"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343105    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f04b95-169f-4b49-8af6-f48c677b6def" containerName="istio-proxy"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343112    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343119    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" containerName="revoke-keys"
Oct 31 16:38:06.343319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.343128    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d9cc78-f6fb-4031-a540-adc0d2e4fadd" containerName="istio-proxy"
Oct 31 16:38:06.347394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.347376    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.396985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.396949    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a1b30a15-openshift-default-dockercfg-cmlh7\""
Oct 31 16:38:06.403421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403393    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.403564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403428    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.403615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403579    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.403672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403614    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.403791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403771    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.403945 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403906    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.404064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.403979    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.404064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.404016    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.404064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.404033    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqvp\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.452569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.452535    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"]
Oct 31 16:38:06.505220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505184    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505221    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505290    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505439    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505635    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505748    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505754    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505788    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505830    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.505897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.505854    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqvp\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.506076 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.506021    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.506169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.506147    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.506247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.506223    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.506335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.506311    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.507760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.507738    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.508067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.508050    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.560269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.560197    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqvp\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.560269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.560207    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:06.661810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:06.661712    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:07.082933 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:07.082900    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a63eb5e_a2ec_4b10_917e_23c984e5d292.slice/crio-231cdf984e2a67a028f7f64c93cf5e63e3297b1a05e9b8c8f4ba09414b833035 WatchSource:0}: Error finding container 231cdf984e2a67a028f7f64c93cf5e63e3297b1a05e9b8c8f4ba09414b833035: Status 404 returned error can't find the container with id 231cdf984e2a67a028f7f64c93cf5e63e3297b1a05e9b8c8f4ba09414b833035
Oct 31 16:38:07.083621 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.083599    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"]
Oct 31 16:38:07.085741 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.085705    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:07.085846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.085784    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:07.085846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.085834    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:07.394860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.394760    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" event={"ID":"7a63eb5e-a2ec-4b10-917e-23c984e5d292","Type":"ContainerStarted","Data":"04b85901aaf1c73a1843c955deb70d6102822f7594f7d53e810060bdf23d2fa8"}
Oct 31 16:38:07.394860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.394832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" event={"ID":"7a63eb5e-a2ec-4b10-917e-23c984e5d292","Type":"ContainerStarted","Data":"231cdf984e2a67a028f7f64c93cf5e63e3297b1a05e9b8c8f4ba09414b833035"}
Oct 31 16:38:07.507828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.507762    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podStartSLOduration=1.50774339 podStartE2EDuration="1.50774339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:07.504873205 +0000 UTC m=+2131.204802518" watchObservedRunningTime="2026-10-31 16:38:07.50774339 +0000 UTC m=+2131.207672704"
Oct 31 16:38:07.662512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:07.662408    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:08.685480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:08.664759    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 31 16:38:08.685480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:08.664816    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 31 16:38:08.712377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:08.708446    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"]
Oct 31 16:38:08.712377 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:08.708679    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" containerID="cri-o://b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89" gracePeriod=30
Oct 31 16:38:09.662821 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:09.662783    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 31 16:38:09.663022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:09.662836    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 31 16:38:09.945602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:09.945516    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"]
Oct 31 16:38:10.000085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.000047    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d0755f-rjgth"]
Oct 31 16:38:10.456510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.456475    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d" path="/var/lib/kubelet/pods/8e8fbfb5-0d06-4f00-b1c2-db7d11ea100d/volumes"
Oct 31 16:38:10.456914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.456876    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"]
Oct 31 16:38:10.505506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.500668    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.505506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.502232    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"]
Oct 31 16:38:10.506538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.506485    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy" containerID="cri-o://9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631" gracePeriod=30
Oct 31 16:38:10.509604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.508005    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"]
Oct 31 16:38:10.551349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.551300    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:38:10.551547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.551356    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:38:10.551547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.551373    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:38:10.551547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.551378    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:38:10.554304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.554281    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gxqh\" (UniqueName: \"kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.554520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.554444    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.554602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.554574    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.656142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.656100    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.657292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.657267    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.659433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.657127    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.660506 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.660483    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gxqh\" (UniqueName: \"kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.662503 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.662256    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 31 16:38:10.662707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.662680    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 31 16:38:10.715053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.710876    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.739300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.739201    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gxqh\" (UniqueName: \"kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh\") pod \"maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:10.841570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:10.841539    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:11.241808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:11.241777    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"]
Oct 31 16:38:11.243667 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:11.243639    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe0e699_4034_4877_98e1_2b7e528bbb41.slice/crio-e1fe9610e7ae7235145f7c1d4c2167bffdf57f9c86a016fe417420594dfab302 WatchSource:0}: Error finding container e1fe9610e7ae7235145f7c1d4c2167bffdf57f9c86a016fe417420594dfab302: Status 404 returned error can't find the container with id e1fe9610e7ae7235145f7c1d4c2167bffdf57f9c86a016fe417420594dfab302
Oct 31 16:38:11.427767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:11.427731    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd" event={"ID":"bbe0e699-4034-4877-98e1-2b7e528bbb41","Type":"ContainerStarted","Data":"e1fe9610e7ae7235145f7c1d4c2167bffdf57f9c86a016fe417420594dfab302"}
Oct 31 16:38:11.662010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:11.661977    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 31 16:38:11.662219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:11.662034    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 31 16:38:11.999027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:11.998993    4519 patch_prober.go:29] interesting pod/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:38:11.999199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:11.999048    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:38:12.437011 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:12.436978    4519 generic.go:356] "Generic (PLEG): container finished" podID="bbe0e699-4034-4877-98e1-2b7e528bbb41" containerID="bca701bbf4a396fa48280a84e5b781fe9f10c9e4a6c2d7b2122cab0814f4a382" exitCode=0
Oct 31 16:38:12.441238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:12.441193    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd" event={"ID":"bbe0e699-4034-4877-98e1-2b7e528bbb41","Type":"ContainerDied","Data":"bca701bbf4a396fa48280a84e5b781fe9f10c9e4a6c2d7b2122cab0814f4a382"}
Oct 31 16:38:12.662684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:12.662603    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Oct 31 16:38:12.662684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:12.662656    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Oct 31 16:38:13.621652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.621617    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:13.706352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.706274    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:13.706727 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.706701    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:13.707605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.707588    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:38:13.727532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.727433    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca\") pod \"bbe0e699-4034-4877-98e1-2b7e528bbb41\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") "
Oct 31 16:38:13.727677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.727595    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh\" (UniqueName: \"kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh\") pod \"bbe0e699-4034-4877-98e1-2b7e528bbb41\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") "
Oct 31 16:38:13.727779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.727759    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca\") pod \"bbe0e699-4034-4877-98e1-2b7e528bbb41\" (UID: \"bbe0e699-4034-4877-98e1-2b7e528bbb41\") "
Oct 31 16:38:13.728548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.728280    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca" pod "bbe0e699-4034-4877-98e1-2b7e528bbb41" (UID: "bbe0e699-4034-4877-98e1-2b7e528bbb41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:13.728548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.728506    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:13.731417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.729918    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca" pod "bbe0e699-4034-4877-98e1-2b7e528bbb41" (UID: "bbe0e699-4034-4877-98e1-2b7e528bbb41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:13.731417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.730903    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh" pod "bbe0e699-4034-4877-98e1-2b7e528bbb41" (UID: "bbe0e699-4034-4877-98e1-2b7e528bbb41"). InnerVolumeSpecName "kube-api-access-9gxqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:13.829659 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.829632    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gxqh\" (UniqueName: \"kubernetes.io/projected/bbe0e699-4034-4877-98e1-2b7e528bbb41-kube-api-access-9gxqh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:13.829843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:13.829816    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe0e699-4034-4877-98e1-2b7e528bbb41-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.001244 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.001213    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:38:14.032640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032611    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.032829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032668    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.032829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032712    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.032829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032764    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.032992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032844    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.032992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032888    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:14.032992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.032980    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.033166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033016    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.033166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033099    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.033297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033169    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket\") pod \"f38d95c8-7437-4a33-8270-1684c798cd86\" (UID: \"f38d95c8-7437-4a33-8270-1684c798cd86\") "
Oct 31 16:38:14.033926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033899    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.034066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033013    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:14.034066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033261    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:14.034066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033672    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:14.034066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.033864    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:14.035756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.035725    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:38:14.035856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.035777    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "kube-api-access-mgnzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:14.035856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.035812    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:14.035994 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.035890    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy" pod "f38d95c8-7437-4a33-8270-1684c798cd86" (UID: "f38d95c8-7437-4a33-8270-1684c798cd86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:14.135617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135579    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135617    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135632    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135646    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135661    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d95c8-7437-4a33-8270-1684c798cd86-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135675    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d95c8-7437-4a33-8270-1684c798cd86-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135689    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgnzx\" (UniqueName: \"kubernetes.io/projected/f38d95c8-7437-4a33-8270-1684c798cd86-kube-api-access-mgnzx\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.135824 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.135705    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d95c8-7437-4a33-8270-1684c798cd86-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:14.454882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.454856    4519 generic.go:356] "Generic (PLEG): container finished" podID="f38d95c8-7437-4a33-8270-1684c798cd86" containerID="b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89" exitCode=0
Oct 31 16:38:14.455010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.454905    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"
Oct 31 16:38:14.455010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.454928    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" event={"ID":"f38d95c8-7437-4a33-8270-1684c798cd86","Type":"ContainerDied","Data":"b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89"}
Oct 31 16:38:14.455010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.454977    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs" event={"ID":"f38d95c8-7437-4a33-8270-1684c798cd86","Type":"ContainerDied","Data":"21fdc6b84c325350342b833dfcb35311a7a1b24b5b316613056132456f116df8"}
Oct 31 16:38:14.455010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.455007    4519 scope.go:122] "RemoveContainer" containerID="b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89"
Oct 31 16:38:14.456489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.456446    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"
Oct 31 16:38:14.456597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.456504    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd" event={"ID":"bbe0e699-4034-4877-98e1-2b7e528bbb41","Type":"ContainerDied","Data":"e1fe9610e7ae7235145f7c1d4c2167bffdf57f9c86a016fe417420594dfab302"}
Oct 31 16:38:14.456597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.456548    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1fe9610e7ae7235145f7c1d4c2167bffdf57f9c86a016fe417420594dfab302"
Oct 31 16:38:14.465991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.465970    4519 scope.go:122] "RemoveContainer" containerID="7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"
Oct 31 16:38:14.475603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.475575    4519 scope.go:122] "RemoveContainer" containerID="b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89"
Oct 31 16:38:14.475893 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:14.475876    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89\": container with ID starting with b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89 not found: ID does not exist" containerID="b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89"
Oct 31 16:38:14.475949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.475901    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89"} err="failed to get container status \"b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89\": rpc error: code = NotFound desc = could not find container \"b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89\": container with ID starting with b23decc050bb35e4e0eeea6a316a796c8d14a70bf78b5bc2dad2ec2a4d75ae89 not found: ID does not exist"
Oct 31 16:38:14.475949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.475918    4519 scope.go:122] "RemoveContainer" containerID="7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"
Oct 31 16:38:14.476112 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:14.476096    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4\": container with ID starting with 7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4 not found: ID does not exist" containerID="7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"
Oct 31 16:38:14.476153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.476116    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4"} err="failed to get container status \"7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4\": rpc error: code = NotFound desc = could not find container \"7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4\": container with ID starting with 7b9ed4f2fc91fa064279773309533710989a8c1f812b18abd94f2c17b25e70a4 not found: ID does not exist"
Oct 31 16:38:14.663163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.663131    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"]
Oct 31 16:38:14.710446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:14.710423    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dac4fa-openshift-default-7c99579dbc-mzdgs"]
Oct 31 16:38:15.801189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.801160    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:38:15.858767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.858737    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.858949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.858834    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.858949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.858868    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859036    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859068    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:15.859124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859083    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859107    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:15.859287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859144    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859251    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859392 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859296    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859392 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859300    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:15.859392 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859361    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs\") pod \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\" (UID: \"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3\") "
Oct 31 16:38:15.859698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859660    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:15.859838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859706    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:15.859996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.859980    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.860047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.860001    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.860047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.860015    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.860047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.860026    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.860047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.860035    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.861804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.861781    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:38:15.861804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.861793    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:15.861929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.861791    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:15.861929 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.861816    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6" pod "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" (UID: "6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3"). InnerVolumeSpecName "kube-api-access-rxzz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:15.961693 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.961659    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzz6\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-kube-api-access-rxzz6\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.961903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.961712    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.961903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.961732    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:15.961903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:15.961747    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:16.448841 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.448809    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" path="/var/lib/kubelet/pods/f38d95c8-7437-4a33-8270-1684c798cd86/volumes"
Oct 31 16:38:16.472832 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.472803    4519 generic.go:356] "Generic (PLEG): container finished" podID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerID="9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631" exitCode=0
Oct 31 16:38:16.472974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.472865    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"
Oct 31 16:38:16.472974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.472887    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" event={"ID":"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3","Type":"ContainerDied","Data":"9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631"}
Oct 31 16:38:16.472974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.472948    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp" event={"ID":"6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3","Type":"ContainerDied","Data":"05bbe83caea16fae2910005b2e3b150eb3b59cb8a87b73b00e32f18f52fb5842"}
Oct 31 16:38:16.473082 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.472978    4519 scope.go:122] "RemoveContainer" containerID="9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631"
Oct 31 16:38:16.483522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.483504    4519 scope.go:122] "RemoveContainer" containerID="9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"
Oct 31 16:38:16.492854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.492834    4519 scope.go:122] "RemoveContainer" containerID="9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631"
Oct 31 16:38:16.493130 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:16.493104    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631\": container with ID starting with 9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631 not found: ID does not exist" containerID="9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631"
Oct 31 16:38:16.493224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.493133    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631"} err="failed to get container status \"9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631\": rpc error: code = NotFound desc = could not find container \"9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631\": container with ID starting with 9c9b03ca1f426b3c72a9942f6d8f507e65752a2830229d0c1d13057bde88d631 not found: ID does not exist"
Oct 31 16:38:16.493224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.493150    4519 scope.go:122] "RemoveContainer" containerID="9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"
Oct 31 16:38:16.493474 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:16.493430    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520\": container with ID starting with 9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520 not found: ID does not exist" containerID="9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"
Oct 31 16:38:16.493525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.493476    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520"} err="failed to get container status \"9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520\": rpc error: code = NotFound desc = could not find container \"9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520\": container with ID starting with 9fc98c09132487471d29a5aa718e6b861bb99d0cdfc13ca6b02c37fd656e3520 not found: ID does not exist"
Oct 31 16:38:16.593304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.593273    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"]
Oct 31 16:38:16.593548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.593520    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy" containerID="cri-o://aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df" gracePeriod=30
Oct 31 16:38:16.766968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.766934    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"]
Oct 31 16:38:16.792241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:16.792205    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-487213-openshift-default-6bbb9d74d7-ffhdp"]
Oct 31 16:38:18.439813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:18.439783    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" path="/var/lib/kubelet/pods/6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3/volumes"
Oct 31 16:38:20.122719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.122683    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"]
Oct 31 16:38:20.123354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123338    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy"
Oct 31 16:38:20.123399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123356    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy"
Oct 31 16:38:20.123399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123375    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bbe0e699-4034-4877-98e1-2b7e528bbb41" containerName="revoke-keys"
Oct 31 16:38:20.123399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123381    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe0e699-4034-4877-98e1-2b7e528bbb41" containerName="revoke-keys"
Oct 31 16:38:20.123399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123398    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy"
Oct 31 16:38:20.123575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123404    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy"
Oct 31 16:38:20.123575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123571    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy"
Oct 31 16:38:20.123671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123581    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy"
Oct 31 16:38:20.123671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123590    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe0e699-4034-4877-98e1-2b7e528bbb41" containerName="revoke-keys"
Oct 31 16:38:20.123671 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.123600    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy"
Oct 31 16:38:20.128661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.128643    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.180105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.180075    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"]
Oct 31 16:38:20.220105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.219353    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.220105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.219427    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5q6\" (UniqueName: \"kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.220105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.219550    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.222197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.222171    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:38:20.222340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.222167    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:38:20.222340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.222274    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:38:20.224931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.224911    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:38:20.321249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.321217    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.321402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.321257    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5q6\" (UniqueName: \"kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.321402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.321297    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.322007 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.321982    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.323558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.323540    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.375040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.374972    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5q6\" (UniqueName: \"kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6\") pod \"maas-api-revoke-sub-39cfef75-shzcb\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.447973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.447941    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:20.816051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:20.816019    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"]
Oct 31 16:38:20.816311 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:20.816277    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377fbaab_69ff_4f9b_980a_92a9a0654976.slice/crio-380dd7883e958c64f3291c915bae0c1da80b6b9f35c6631a43b9d7cf2363a421 WatchSource:0}: Error finding container 380dd7883e958c64f3291c915bae0c1da80b6b9f35c6631a43b9d7cf2363a421: Status 404 returned error can't find the container with id 380dd7883e958c64f3291c915bae0c1da80b6b9f35c6631a43b9d7cf2363a421
Oct 31 16:38:21.472585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.472552    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"]
Oct 31 16:38:21.473190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.473174    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy"
Oct 31 16:38:21.473245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.473193    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy"
Oct 31 16:38:21.473245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.473228    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy"
Oct 31 16:38:21.473245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.473233    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38d95c8-7437-4a33-8270-1684c798cd86" containerName="istio-proxy"
Oct 31 16:38:21.473358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.473349    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d85ad72-dbb1-4cf1-bd4d-82746fac3eb3" containerName="istio-proxy"
Oct 31 16:38:21.477180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.477161    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.521190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.521154    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb" event={"ID":"377fbaab-69ff-4f9b-980a-92a9a0654976","Type":"ContainerStarted","Data":"b1fe9841afd00693fedc454e0bda3b1c6db7062f480599ef723b9838e8ad2c3c"}
Oct 31 16:38:21.521339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.521218    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb" event={"ID":"377fbaab-69ff-4f9b-980a-92a9a0654976","Type":"ContainerStarted","Data":"380dd7883e958c64f3291c915bae0c1da80b6b9f35c6631a43b9d7cf2363a421"}
Oct 31 16:38:21.524069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.524048    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"]
Oct 31 16:38:21.528360 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.528341    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.536227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536202    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536236    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536263    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536279    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536346    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgh5f\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536466    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536540    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536576    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.536766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.536742    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.571389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.571356    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"]
Oct 31 16:38:21.618341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.618305    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"]
Oct 31 16:38:21.637894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.637864    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.637904    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.637934    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638041    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638081    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgh5f\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638126    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638275    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638305    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638329    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.638520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638426    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.638520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638474    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.638590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638524    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.638590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638530    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638568    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvlbp\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.638590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638568    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638727    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.638766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638759    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638817    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.638964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638944    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.639020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.638971    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.639079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.639021    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.639079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.639056    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.640612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.640587    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.641159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.641130    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.696733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.696701    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.698716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.698685    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgh5f\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.740837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740761    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.740837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740803    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.740837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740833    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740864    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvlbp\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740897    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740941    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.740967    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741107 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741015    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741194    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741269    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741491    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741595    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741640    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.741734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.741707    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.743797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.743778    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.743886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.743781    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.792715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.791962    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:21.796932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.796060    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvlbp\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.796932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.796857    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.804719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.804669    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb" podStartSLOduration=1.804657013 podStartE2EDuration="1.804657013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:21.695896528 +0000 UTC m=+2145.395825843" watchObservedRunningTime="2026-10-31 16:38:21.804657013 +0000 UTC m=+2145.504586323"
Oct 31 16:38:21.845750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.845702    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:21.869044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.869023    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:38:21.946710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.946680    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.946834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.946781    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.946897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.946834    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.946943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.946897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.946982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.946955    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.947020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.947003    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.947054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.947020    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.947109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.947089    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.947235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.947142    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy\") pod \"884f7246-858e-42ac-93d0-97e24b88c8ea\" (UID: \"884f7246-858e-42ac-93d0-97e24b88c8ea\") "
Oct 31 16:38:21.947713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.947680    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:21.947889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.947869    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:21.948193 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.948055    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:21.948291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.948116    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:21.948291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.948165    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:21.948291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.948214    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:21.950387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.950360    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:21.950521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.950500    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "kube-api-access-npb2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:21.951767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.951746    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:38:21.951834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:21.951770    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy" pod "884f7246-858e-42ac-93d0-97e24b88c8ea" (UID: "884f7246-858e-42ac-93d0-97e24b88c8ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049339    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049371    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049383    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npb2m\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-kube-api-access-npb2m\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049392    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884f7246-858e-42ac-93d0-97e24b88c8ea-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049401    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049409    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049418    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884f7246-858e-42ac-93d0-97e24b88c8ea-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.049422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.049426    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884f7246-858e-42ac-93d0-97e24b88c8ea-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:22.223656 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:22.223626    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8350bd2c_9c50_4fb6_bd89_28bab01a2aed.slice/crio-e6af621761a56b4cbc5123996faf6cbaa8917b1911941a4262aba61258c557b1 WatchSource:0}: Error finding container e6af621761a56b4cbc5123996faf6cbaa8917b1911941a4262aba61258c557b1: Status 404 returned error can't find the container with id e6af621761a56b4cbc5123996faf6cbaa8917b1911941a4262aba61258c557b1
Oct 31 16:38:22.225337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.225308    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"]
Oct 31 16:38:22.226951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.226919    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.227062 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.226997    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.227132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.227066    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.279386 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:22.277965    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d6b028_b826_4c73_8625_2c3291adc268.slice/crio-e9c34174aaf2db9ecf4dba9a358c131086a7b4666c885b4ccb9c29367f5413b8 WatchSource:0}: Error finding container e9c34174aaf2db9ecf4dba9a358c131086a7b4666c885b4ccb9c29367f5413b8: Status 404 returned error can't find the container with id e9c34174aaf2db9ecf4dba9a358c131086a7b4666c885b4ccb9c29367f5413b8
Oct 31 16:38:22.281087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.281041    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.281202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.281142    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.281260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.281212    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.288606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.287659    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"]
Oct 31 16:38:22.532389 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.532353    4519 generic.go:356] "Generic (PLEG): container finished" podID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerID="aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df" exitCode=0
Oct 31 16:38:22.532893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.532412    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"
Oct 31 16:38:22.532893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.532437    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" event={"ID":"884f7246-858e-42ac-93d0-97e24b88c8ea","Type":"ContainerDied","Data":"aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df"}
Oct 31 16:38:22.532893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.532522    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb" event={"ID":"884f7246-858e-42ac-93d0-97e24b88c8ea","Type":"ContainerDied","Data":"68ca8b94fb9773a62a5f5d72f617c5440d6c00c8a6510feb5caf701c4fbe5d30"}
Oct 31 16:38:22.532893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.532557    4519 scope.go:122] "RemoveContainer" containerID="aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df"
Oct 31 16:38:22.535147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.534775    4519 generic.go:356] "Generic (PLEG): container finished" podID="377fbaab-69ff-4f9b-980a-92a9a0654976" containerID="b1fe9841afd00693fedc454e0bda3b1c6db7062f480599ef723b9838e8ad2c3c" exitCode=0
Oct 31 16:38:22.535147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.534904    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb" event={"ID":"377fbaab-69ff-4f9b-980a-92a9a0654976","Type":"ContainerDied","Data":"b1fe9841afd00693fedc454e0bda3b1c6db7062f480599ef723b9838e8ad2c3c"}
Oct 31 16:38:22.537709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.537686    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" event={"ID":"e0d6b028-b826-4c73-8625-2c3291adc268","Type":"ContainerStarted","Data":"e245c6d45b707b85b8e3e1b28e812f82a26eaa808d559a0fb20d759164a0f7ac"}
Oct 31 16:38:22.537835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.537735    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" event={"ID":"e0d6b028-b826-4c73-8625-2c3291adc268","Type":"ContainerStarted","Data":"e9c34174aaf2db9ecf4dba9a358c131086a7b4666c885b4ccb9c29367f5413b8"}
Oct 31 16:38:22.544718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.544692    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" event={"ID":"8350bd2c-9c50-4fb6-bd89-28bab01a2aed","Type":"ContainerStarted","Data":"886177cbfe060752d887d7ecf70f37ce6d1b0456cd86e2996889bbf155e12d8a"}
Oct 31 16:38:22.544820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.544740    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" event={"ID":"8350bd2c-9c50-4fb6-bd89-28bab01a2aed","Type":"ContainerStarted","Data":"e6af621761a56b4cbc5123996faf6cbaa8917b1911941a4262aba61258c557b1"}
Oct 31 16:38:22.547963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.547943    4519 scope.go:122] "RemoveContainer" containerID="688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"
Oct 31 16:38:22.560238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.560215    4519 scope.go:122] "RemoveContainer" containerID="aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df"
Oct 31 16:38:22.569895 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:22.569851    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df\": container with ID starting with aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df not found: ID does not exist" containerID="aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df"
Oct 31 16:38:22.570036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.569912    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df"} err="failed to get container status \"aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df\": rpc error: code = NotFound desc = could not find container \"aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df\": container with ID starting with aa96e7348ddb3f1be3fc42e2b2f9dd7df65bdc60432640529fac1bb34ff067df not found: ID does not exist"
Oct 31 16:38:22.570036 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.569949    4519 scope.go:122] "RemoveContainer" containerID="688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"
Oct 31 16:38:22.570784 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:22.570750    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551\": container with ID starting with 688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551 not found: ID does not exist" containerID="688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"
Oct 31 16:38:22.570949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.570795    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551"} err="failed to get container status \"688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551\": rpc error: code = NotFound desc = could not find container \"688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551\": container with ID starting with 688edb4ecacf337e234655897fbbf4abb7b2fdb52f77c6de5a0b1d59b12ad551 not found: ID does not exist"
Oct 31 16:38:22.698865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.698789    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podStartSLOduration=1.69877078 podStartE2EDuration="1.69877078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:22.696984378 +0000 UTC m=+2146.396913690" watchObservedRunningTime="2026-10-31 16:38:22.69877078 +0000 UTC m=+2146.398700095"
Oct 31 16:38:22.793315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.793284    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:22.806778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.806739    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"]
Oct 31 16:38:22.810346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.810289    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d0755f-openshift-default-69b4777c4c-qrzxb"]
Oct 31 16:38:22.846047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:22.846013    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:23.029031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:23.028969    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podStartSLOduration=2.028951197 podStartE2EDuration="2.028951197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:23.024701299 +0000 UTC m=+2146.724630611" watchObservedRunningTime="2026-10-31 16:38:23.028951197 +0000 UTC m=+2146.728880507"
Oct 31 16:38:23.825380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:23.825295    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:38:23.825380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:23.825355    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:38:23.848151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:23.847620    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 16:38:23.848151 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:23.847674    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 16:38:23.932913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:23.932881    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:24.029600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.029543    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca\") pod \"377fbaab-69ff-4f9b-980a-92a9a0654976\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") "
Oct 31 16:38:24.029931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.029904    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca" pod "377fbaab-69ff-4f9b-980a-92a9a0654976" (UID: "377fbaab-69ff-4f9b-980a-92a9a0654976"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:24.030138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.030118    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca\") pod \"377fbaab-69ff-4f9b-980a-92a9a0654976\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") "
Oct 31 16:38:24.030556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.030536    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6\" (UniqueName: \"kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6\") pod \"377fbaab-69ff-4f9b-980a-92a9a0654976\" (UID: \"377fbaab-69ff-4f9b-980a-92a9a0654976\") "
Oct 31 16:38:24.031169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.031150    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:24.034040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.034001    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca" pod "377fbaab-69ff-4f9b-980a-92a9a0654976" (UID: "377fbaab-69ff-4f9b-980a-92a9a0654976"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:24.034187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.034165    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6" pod "377fbaab-69ff-4f9b-980a-92a9a0654976" (UID: "377fbaab-69ff-4f9b-980a-92a9a0654976"). InnerVolumeSpecName "kube-api-access-4h5q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:24.043441 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:24.043413    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:38:24.138206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.133568    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/377fbaab-69ff-4f9b-980a-92a9a0654976-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:24.138206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.133602    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5q6\" (UniqueName: \"kubernetes.io/projected/377fbaab-69ff-4f9b-980a-92a9a0654976-kube-api-access-4h5q6\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:24.440349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.440271    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" path="/var/lib/kubelet/pods/884f7246-858e-42ac-93d0-97e24b88c8ea/volumes"
Oct 31 16:38:24.622417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.622387    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"
Oct 31 16:38:24.622617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.622433    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb" event={"ID":"377fbaab-69ff-4f9b-980a-92a9a0654976","Type":"ContainerDied","Data":"380dd7883e958c64f3291c915bae0c1da80b6b9f35c6631a43b9d7cf2363a421"}
Oct 31 16:38:24.622617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.622503    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="380dd7883e958c64f3291c915bae0c1da80b6b9f35c6631a43b9d7cf2363a421"
Oct 31 16:38:24.794479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.794049    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:38:24.794479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.794120    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:38:24.847021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.846937    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 16:38:24.847021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:24.847002    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 16:38:25.793479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:25.792901    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:38:25.793479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:25.792979    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:38:25.846902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:25.846873    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 16:38:25.847140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:25.847118    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 16:38:26.792946 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:26.792905    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:38:26.793245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:26.793212    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:38:26.846644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:26.846606    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 16:38:26.846835 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:26.846676    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 16:38:27.804478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:27.801909    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:38:27.804478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:27.801991    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:38:27.846235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:27.846198    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 16:38:27.846392 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:27.846250    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 16:38:28.794057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:28.794011    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:38:28.794330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:28.794302    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:38:28.846869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:28.846835    4519 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 16:38:28.854651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:28.854582    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 16:38:29.133201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.133121    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:38:29.797396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.797363    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:29.797791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.797769    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:29.798370 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.798351    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:38:29.851176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.851142    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:29.851655 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.851482    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:29.852301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:29.852279    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:38:52.686416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:52.686385    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"]
Oct 31 16:38:52.733480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:52.733426    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-705a67-66slk"]
Oct 31 16:38:53.083688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.083656    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"]
Oct 31 16:38:53.084283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084270    4519 cpu_manager.go:414] "containerMap: removing container" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy"
Oct 31 16:38:53.084337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084285    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy"
Oct 31 16:38:53.084337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084301    4519 cpu_manager.go:414] "containerMap: removing container" podUID="377fbaab-69ff-4f9b-980a-92a9a0654976" containerName="revoke-keys"
Oct 31 16:38:53.084337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084306    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377fbaab-69ff-4f9b-980a-92a9a0654976" containerName="revoke-keys"
Oct 31 16:38:53.084434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084417    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy"
Oct 31 16:38:53.084434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084427    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="377fbaab-69ff-4f9b-980a-92a9a0654976" containerName="revoke-keys"
Oct 31 16:38:53.084538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.084434    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy"
Oct 31 16:38:53.089689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.089669    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.140175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.140142    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"]
Oct 31 16:38:53.183518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.183478    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:38:53.183518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.183478    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:38:53.183518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.183519    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:38:53.183775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.183621    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:38:53.189207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.189180    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.189301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.189210    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.189393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.189376    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2mb\" (UniqueName: \"kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.290665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.290629    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.290665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.290666    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.290925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.290816    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2mb\" (UniqueName: \"kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.291320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.291295    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.293021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.292998    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.345096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.345025    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2mb\" (UniqueName: \"kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb\") pod \"maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.403838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.403807    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:53.772434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.772406    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"]
Oct 31 16:38:53.772979 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:53.772947    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1f2769_2825_43fd_9324_53992558b371.slice/crio-dc2b13901e0edf50195a7cef999fce74b41a78c85aab79fa0d8dfb7c7619c839 WatchSource:0}: Error finding container dc2b13901e0edf50195a7cef999fce74b41a78c85aab79fa0d8dfb7c7619c839: Status 404 returned error can't find the container with id dc2b13901e0edf50195a7cef999fce74b41a78c85aab79fa0d8dfb7c7619c839
Oct 31 16:38:53.861974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:53.861943    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj" event={"ID":"3c1f2769-2825-43fd-9324-53992558b371","Type":"ContainerStarted","Data":"dc2b13901e0edf50195a7cef999fce74b41a78c85aab79fa0d8dfb7c7619c839"}
Oct 31 16:38:54.006918 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:38:54.006879    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:38:54.440675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:54.440599    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ccf1b4a-4e0b-47dd-a08e-3241f00d6154" path="/var/lib/kubelet/pods/3ccf1b4a-4e0b-47dd-a08e-3241f00d6154/volumes"
Oct 31 16:38:54.871104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:54.871069    4519 generic.go:356] "Generic (PLEG): container finished" podID="3c1f2769-2825-43fd-9324-53992558b371" containerID="598d08f18b3fb67243ef34cb6b114555885befbc5d1cb1c586e10c5be3d0546c" exitCode=0
Oct 31 16:38:54.871530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:54.871145    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj" event={"ID":"3c1f2769-2825-43fd-9324-53992558b371","Type":"ContainerDied","Data":"598d08f18b3fb67243ef34cb6b114555885befbc5d1cb1c586e10c5be3d0546c"}
Oct 31 16:38:56.061248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.061223    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:56.083172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.083145    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"]
Oct 31 16:38:56.083848 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.083828    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3c1f2769-2825-43fd-9324-53992558b371" containerName="revoke-keys"
Oct 31 16:38:56.083960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.083849    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1f2769-2825-43fd-9324-53992558b371" containerName="revoke-keys"
Oct 31 16:38:56.083960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.083922    4519 cpu_manager.go:414] "containerMap: removing container" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy"
Oct 31 16:38:56.083960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.083931    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884f7246-858e-42ac-93d0-97e24b88c8ea" containerName="istio-proxy"
Oct 31 16:38:56.084141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.084126    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1f2769-2825-43fd-9324-53992558b371" containerName="revoke-keys"
Oct 31 16:38:56.087593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.087574    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.130217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.130156    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca\") pod \"3c1f2769-2825-43fd-9324-53992558b371\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") "
Oct 31 16:38:56.130353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.130213    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca\") pod \"3c1f2769-2825-43fd-9324-53992558b371\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") "
Oct 31 16:38:56.130353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.130254    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb\" (UniqueName: \"kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb\") pod \"3c1f2769-2825-43fd-9324-53992558b371\" (UID: \"3c1f2769-2825-43fd-9324-53992558b371\") "
Oct 31 16:38:56.130557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.130536    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca" pod "3c1f2769-2825-43fd-9324-53992558b371" (UID: "3c1f2769-2825-43fd-9324-53992558b371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:56.130777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.130759    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.130987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.130961    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.131102 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.131054    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbrl\" (UniqueName: \"kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.131170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.131152    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f2769-2825-43fd-9324-53992558b371-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:56.132891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.132871    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca" pod "3c1f2769-2825-43fd-9324-53992558b371" (UID: "3c1f2769-2825-43fd-9324-53992558b371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:56.132971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.132905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb" pod "3c1f2769-2825-43fd-9324-53992558b371" (UID: "3c1f2769-2825-43fd-9324-53992558b371"). InnerVolumeSpecName "kube-api-access-7g2mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:56.141713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.141689    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"]
Oct 31 16:38:56.232127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.232089    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.232320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.232255    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.232400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.232345    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbrl\" (UniqueName: \"kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.232562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.232545    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1f2769-2825-43fd-9324-53992558b371-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:56.232632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.232567    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2mb\" (UniqueName: \"kubernetes.io/projected/3c1f2769-2825-43fd-9324-53992558b371-kube-api-access-7g2mb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:56.232911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.232891    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.234573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.234549    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.286369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.286338    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbrl\" (UniqueName: \"kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl\") pod \"maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.401359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.401285    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:56.769733 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:38:56.769691    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abafe09_6ecb_4212_a595_6f1279746c61.slice/crio-94338f52da02ce0f5af5751a7ea67ed3985a070ab28733e7a96493dfd2c10d4a WatchSource:0}: Error finding container 94338f52da02ce0f5af5751a7ea67ed3985a070ab28733e7a96493dfd2c10d4a: Status 404 returned error can't find the container with id 94338f52da02ce0f5af5751a7ea67ed3985a070ab28733e7a96493dfd2c10d4a
Oct 31 16:38:56.770245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.770225    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"]
Oct 31 16:38:56.886411 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.886373    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw" event={"ID":"0abafe09-6ecb-4212-a595-6f1279746c61","Type":"ContainerStarted","Data":"94338f52da02ce0f5af5751a7ea67ed3985a070ab28733e7a96493dfd2c10d4a"}
Oct 31 16:38:56.887850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.887831    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"
Oct 31 16:38:56.887960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.887832    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj" event={"ID":"3c1f2769-2825-43fd-9324-53992558b371","Type":"ContainerDied","Data":"dc2b13901e0edf50195a7cef999fce74b41a78c85aab79fa0d8dfb7c7619c839"}
Oct 31 16:38:56.888013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:56.887958    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2b13901e0edf50195a7cef999fce74b41a78c85aab79fa0d8dfb7c7619c839"
Oct 31 16:38:57.896312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:57.896235    4519 generic.go:356] "Generic (PLEG): container finished" podID="0abafe09-6ecb-4212-a595-6f1279746c61" containerID="861a86794e6860d8d320ed92fbe83c2af663f62e601c8f3439ec24c7e0eb9aee" exitCode=0
Oct 31 16:38:57.896726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:57.896323    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw" event={"ID":"0abafe09-6ecb-4212-a595-6f1279746c61","Type":"ContainerDied","Data":"861a86794e6860d8d320ed92fbe83c2af663f62e601c8f3439ec24c7e0eb9aee"}
Oct 31 16:38:59.059804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.059773    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:59.181365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.180357    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca\") pod \"0abafe09-6ecb-4212-a595-6f1279746c61\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") "
Oct 31 16:38:59.181365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.180623    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl\" (UniqueName: \"kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl\") pod \"0abafe09-6ecb-4212-a595-6f1279746c61\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") "
Oct 31 16:38:59.181365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.180810    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca\") pod \"0abafe09-6ecb-4212-a595-6f1279746c61\" (UID: \"0abafe09-6ecb-4212-a595-6f1279746c61\") "
Oct 31 16:38:59.181365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.180890    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca" pod "0abafe09-6ecb-4212-a595-6f1279746c61" (UID: "0abafe09-6ecb-4212-a595-6f1279746c61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:59.182530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.182494    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:59.184431 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.184402    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl" pod "0abafe09-6ecb-4212-a595-6f1279746c61" (UID: "0abafe09-6ecb-4212-a595-6f1279746c61"). InnerVolumeSpecName "kube-api-access-8qbrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:59.188769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.188738    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca" pod "0abafe09-6ecb-4212-a595-6f1279746c61" (UID: "0abafe09-6ecb-4212-a595-6f1279746c61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:59.283959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.283925    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0abafe09-6ecb-4212-a595-6f1279746c61-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:59.283959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.283956    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbrl\" (UniqueName: \"kubernetes.io/projected/0abafe09-6ecb-4212-a595-6f1279746c61-kube-api-access-8qbrl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:59.913548 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.913518    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"
Oct 31 16:38:59.913722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.913544    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw" event={"ID":"0abafe09-6ecb-4212-a595-6f1279746c61","Type":"ContainerDied","Data":"94338f52da02ce0f5af5751a7ea67ed3985a070ab28733e7a96493dfd2c10d4a"}
Oct 31 16:38:59.913722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:38:59.913592    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94338f52da02ce0f5af5751a7ea67ed3985a070ab28733e7a96493dfd2c10d4a"
Oct 31 16:39:00.380675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.380643    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"]
Oct 31 16:39:00.381277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.381264    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0abafe09-6ecb-4212-a595-6f1279746c61" containerName="revoke-keys"
Oct 31 16:39:00.381338 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.381278    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0abafe09-6ecb-4212-a595-6f1279746c61" containerName="revoke-keys"
Oct 31 16:39:00.381420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.381411    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0abafe09-6ecb-4212-a595-6f1279746c61" containerName="revoke-keys"
Oct 31 16:39:00.385220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.385199    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.435469 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.435425    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:39:00.441438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.441413    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"]
Oct 31 16:39:00.481215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.481192    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:39:00.481361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.481224    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:39:00.481361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.481242    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:39:00.513596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.513567    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.513596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.513597    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.513782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.513754    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwjz\" (UniqueName: \"kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.615543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.615509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.615543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.615553    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.615763 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.615669    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwjz\" (UniqueName: \"kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.616232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.616210    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.618018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.617993    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.678310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.678241    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwjz\" (UniqueName: \"kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:00.700432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:00.700408    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:01.066119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:01.066093    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"]
Oct 31 16:39:01.066565 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:39:01.066540    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125915c2_a89e_4ba0_9bdf_cb93c3f75b7b.slice/crio-3b823c8323ab9f0a9cf075363f4d1e9d00cb20e63cd32248ae45a888bcddfd30 WatchSource:0}: Error finding container 3b823c8323ab9f0a9cf075363f4d1e9d00cb20e63cd32248ae45a888bcddfd30: Status 404 returned error can't find the container with id 3b823c8323ab9f0a9cf075363f4d1e9d00cb20e63cd32248ae45a888bcddfd30
Oct 31 16:39:01.930358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:01.930316    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8" event={"ID":"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b","Type":"ContainerStarted","Data":"b0137266447d16317ed093f4e80e782dd9a7c7dc8b4d019ad6c9b504413f1d6e"}
Oct 31 16:39:01.930784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:01.930376    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8" event={"ID":"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b","Type":"ContainerStarted","Data":"3b823c8323ab9f0a9cf075363f4d1e9d00cb20e63cd32248ae45a888bcddfd30"}
Oct 31 16:39:02.038910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:02.038864    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8" podStartSLOduration=2.038851187 podStartE2EDuration="2.038851187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:39:02.036754321 +0000 UTC m=+2185.736683653" watchObservedRunningTime="2026-10-31 16:39:02.038851187 +0000 UTC m=+2185.738780498"
Oct 31 16:39:02.939248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:02.939216    4519 generic.go:356] "Generic (PLEG): container finished" podID="125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" containerID="b0137266447d16317ed093f4e80e782dd9a7c7dc8b4d019ad6c9b504413f1d6e" exitCode=0
Oct 31 16:39:02.939716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:02.939307    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8" event={"ID":"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b","Type":"ContainerDied","Data":"b0137266447d16317ed093f4e80e782dd9a7c7dc8b4d019ad6c9b504413f1d6e"}
Oct 31 16:39:04.117869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.117842    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:04.267887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.267846    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca\") pod \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") "
Oct 31 16:39:04.268165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.268135    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca\") pod \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") "
Oct 31 16:39:04.268243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.268178    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca" pod "125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" (UID: "125915c2-a89e-4ba0-9bdf-cb93c3f75b7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:04.268243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.268200    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz\" (UniqueName: \"kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz\") pod \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\" (UID: \"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b\") "
Oct 31 16:39:04.268892 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.268865    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.270309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.270286    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca" pod "125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" (UID: "125915c2-a89e-4ba0-9bdf-cb93c3f75b7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:04.270410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.270368    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz" pod "125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" (UID: "125915c2-a89e-4ba0-9bdf-cb93c3f75b7b"). InnerVolumeSpecName "kube-api-access-9dwjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:04.370444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.370410    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.370444 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.370439    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwjz\" (UniqueName: \"kubernetes.io/projected/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b-kube-api-access-9dwjz\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.851515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.851481    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"]
Oct 31 16:39:04.925050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.925012    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"]
Oct 31 16:39:04.929445 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.929413    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"]
Oct 31 16:39:04.959440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.959402    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8" event={"ID":"125915c2-a89e-4ba0-9bdf-cb93c3f75b7b","Type":"ContainerDied","Data":"3b823c8323ab9f0a9cf075363f4d1e9d00cb20e63cd32248ae45a888bcddfd30"}
Oct 31 16:39:04.959662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.959440    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"
Oct 31 16:39:04.959662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:04.959499    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b823c8323ab9f0a9cf075363f4d1e9d00cb20e63cd32248ae45a888bcddfd30"
Oct 31 16:39:11.789042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:11.789005    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"]
Oct 31 16:39:11.893607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:11.893572    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"]
Oct 31 16:39:13.045163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:13.045128    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"]
Oct 31 16:39:13.048526 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:13.048503    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e5c1e77-82nwx"]
Oct 31 16:39:13.774429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:13.774397    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"]
Oct 31 16:39:13.821311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:13.821280    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"]
Oct 31 16:39:13.821520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:13.821494    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" containerID="cri-o://f3ba42ee3f6666586be3ee7e3b5ea83be222bad5646e8602535957c6514a9c82" gracePeriod=30
Oct 31 16:39:14.042128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:14.042026    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy" containerID="cri-o://87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010" gracePeriod=30
Oct 31 16:39:14.440875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:14.440792    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290b5587-0512-44c5-8e6a-00634ee620bb" path="/var/lib/kubelet/pods/290b5587-0512-44c5-8e6a-00634ee620bb/volumes"
Oct 31 16:39:16.789511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:16.789439    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerName="payload-processing" containerID="cri-o://de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27" gracePeriod=25
Oct 31 16:39:16.893999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:16.893959    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerName="payload-pre-processing" containerID="cri-o://764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c" gracePeriod=25
Oct 31 16:39:17.866732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:17.866647    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:39:18.273421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:18.273389    4519 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:39:18.273600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:18.273468    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:39:19.128129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.127828    4519 generic.go:356] "Generic (PLEG): container finished" podID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerID="f3ba42ee3f6666586be3ee7e3b5ea83be222bad5646e8602535957c6514a9c82" exitCode=0
Oct 31 16:39:19.128129 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.127989    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" event={"ID":"bc3c224c-3bdd-410a-b0bc-4ff880c613a3","Type":"ContainerDied","Data":"f3ba42ee3f6666586be3ee7e3b5ea83be222bad5646e8602535957c6514a9c82"}
Oct 31 16:39:19.216955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.216614    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:39:19.336940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.336899    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337106    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337163    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337200    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337351 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337326    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337364    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337409    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337442    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.337598 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.337493    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo\") pod \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\" (UID: \"bc3c224c-3bdd-410a-b0bc-4ff880c613a3\") "
Oct 31 16:39:19.339897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.338818    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.339897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.339044    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.339897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.339278    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.339897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.339735    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:19.339897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.339867    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.341218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.341193    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:19.342197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.342162    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "kube-api-access-9dfzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.342809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.342783    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.347872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.347844    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy" pod "bc3c224c-3bdd-410a-b0bc-4ff880c613a3" (UID: "bc3c224c-3bdd-410a-b0bc-4ff880c613a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.367158 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.367136    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439876    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439910    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439921    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439931    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439940    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439950    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439958    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dfzn\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-kube-api-access-9dfzn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.439963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439969    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.440341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.439977    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3c224c-3bdd-410a-b0bc-4ff880c613a3-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.541132 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541087    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541184    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541266    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541311    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541519    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.541624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541574    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541714    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.541846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541826    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541862    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.541941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541827    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:19.541941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541902    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.542015 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.541986    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy\") pod \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\" (UID: \"4b8ee24f-7d6e-445f-88f4-08bbf07da809\") "
Oct 31 16:39:19.542435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542189    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.542435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542356    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.542925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542898    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.543016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542931    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.543016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542947    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.543016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542962    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.543016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.542976    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.544803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.544779    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:19.544904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.544811    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "kube-api-access-mgvfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.544904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.544852    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.546667 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.546637    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy" pod "4b8ee24f-7d6e-445f-88f4-08bbf07da809" (UID: "4b8ee24f-7d6e-445f-88f4-08bbf07da809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.644173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.644140    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.644173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.644173    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.644173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.644187    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8ee24f-7d6e-445f-88f4-08bbf07da809-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.644417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.644202    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvfv\" (UniqueName: \"kubernetes.io/projected/4b8ee24f-7d6e-445f-88f4-08bbf07da809-kube-api-access-mgvfv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.883099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.883047    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main" containerID="cri-o://b3b5efc5391874f7566c1359ddfb620d96eac7c6a060a7cda9feb4563c37e406" gracePeriod=45
Oct 31 16:39:19.949694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.949651    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main" containerID="cri-o://274281c410757ec16eaa5392fdb2ddb8cb06f8123dbdcba5d576d047f71c0a3e" gracePeriod=45
Oct 31 16:39:19.954211 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:19.954174    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main" containerID="cri-o://69fe5be07cfe39afbf3bd1d3dc88f4d8b0f43f83e5cd8023c8b5a5cab7fa9ab4" gracePeriod=45
Oct 31 16:39:20.174058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.173973    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerID="87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010" exitCode=0
Oct 31 16:39:20.174427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.174129    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" event={"ID":"4b8ee24f-7d6e-445f-88f4-08bbf07da809","Type":"ContainerDied","Data":"87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010"}
Oct 31 16:39:20.174427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.174183    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6" event={"ID":"4b8ee24f-7d6e-445f-88f4-08bbf07da809","Type":"ContainerDied","Data":"c7d98e5cfd1512a6210b166039af73720a44eb8b9945f93960de064385122015"}
Oct 31 16:39:20.174427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.174212    4519 scope.go:122] "RemoveContainer" containerID="87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010"
Oct 31 16:39:20.174427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.174405    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"
Oct 31 16:39:20.183863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.183660    4519 generic.go:356] "Generic (PLEG): container finished" podID="3b495c62-d368-4270-9576-f59625676332" containerID="69fe5be07cfe39afbf3bd1d3dc88f4d8b0f43f83e5cd8023c8b5a5cab7fa9ab4" exitCode=0
Oct 31 16:39:20.183863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.183783    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" event={"ID":"3b495c62-d368-4270-9576-f59625676332","Type":"ContainerDied","Data":"69fe5be07cfe39afbf3bd1d3dc88f4d8b0f43f83e5cd8023c8b5a5cab7fa9ab4"}
Oct 31 16:39:20.186619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.186298    4519 generic.go:356] "Generic (PLEG): container finished" podID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerID="b3b5efc5391874f7566c1359ddfb620d96eac7c6a060a7cda9feb4563c37e406" exitCode=0
Oct 31 16:39:20.186619 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.186501    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" event={"ID":"372f17d9-af5d-4e0c-9ac5-25ee9daffa81","Type":"ContainerDied","Data":"b3b5efc5391874f7566c1359ddfb620d96eac7c6a060a7cda9feb4563c37e406"}
Oct 31 16:39:20.188965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.188835    4519 generic.go:356] "Generic (PLEG): container finished" podID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerID="274281c410757ec16eaa5392fdb2ddb8cb06f8123dbdcba5d576d047f71c0a3e" exitCode=0
Oct 31 16:39:20.190374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.188940    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" event={"ID":"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd","Type":"ContainerDied","Data":"274281c410757ec16eaa5392fdb2ddb8cb06f8123dbdcba5d576d047f71c0a3e"}
Oct 31 16:39:20.197717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.194097    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n" event={"ID":"bc3c224c-3bdd-410a-b0bc-4ff880c613a3","Type":"ContainerDied","Data":"7e0f38d9b52dd7f7597c25fe8ddf5caed27df1f599cf5c52ce7e1b1cc737a89f"}
Oct 31 16:39:20.197717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.194212    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"
Oct 31 16:39:20.218265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.217970    4519 scope.go:122] "RemoveContainer" containerID="87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010"
Oct 31 16:39:20.219862 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:20.218512    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010\": container with ID starting with 87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010 not found: ID does not exist" containerID="87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010"
Oct 31 16:39:20.219862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.218546    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010"} err="failed to get container status \"87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010\": rpc error: code = NotFound desc = could not find container \"87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010\": container with ID starting with 87e0bf8ce5896d4c11c922744e34aefd55389ca7263973a0673ffbb6f0b3e010 not found: ID does not exist"
Oct 31 16:39:20.219862 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.218568    4519 scope.go:122] "RemoveContainer" containerID="f3ba42ee3f6666586be3ee7e3b5ea83be222bad5646e8602535957c6514a9c82"
Oct 31 16:39:20.265541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.265513    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:39:20.272450 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.272417    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm\") pod \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") "
Oct 31 16:39:20.272972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.272630    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home\") pod \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") "
Oct 31 16:39:20.272972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.272675    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir\") pod \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") "
Oct 31 16:39:20.272972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.272777    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs\" (UniqueName: \"kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs\") pod \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") "
Oct 31 16:39:20.272972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.272831    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache\") pod \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") "
Oct 31 16:39:20.272972 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.272883    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg\" (UniqueName: \"kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg\") pod \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\" (UID: \"372f17d9-af5d-4e0c-9ac5-25ee9daffa81\") "
Oct 31 16:39:20.273766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.273567    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir" pod "372f17d9-af5d-4e0c-9ac5-25ee9daffa81" (UID: "372f17d9-af5d-4e0c-9ac5-25ee9daffa81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.273971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.273909    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home" pod "372f17d9-af5d-4e0c-9ac5-25ee9daffa81" (UID: "372f17d9-af5d-4e0c-9ac5-25ee9daffa81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.274702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.274635    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache" pod "372f17d9-af5d-4e0c-9ac5-25ee9daffa81" (UID: "372f17d9-af5d-4e0c-9ac5-25ee9daffa81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.276809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.276775    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs" pod "372f17d9-af5d-4e0c-9ac5-25ee9daffa81" (UID: "372f17d9-af5d-4e0c-9ac5-25ee9daffa81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:20.278144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.278103    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg" pod "372f17d9-af5d-4e0c-9ac5-25ee9daffa81" (UID: "372f17d9-af5d-4e0c-9ac5-25ee9daffa81"). InnerVolumeSpecName "kube-api-access-79gvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:20.279274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.279220    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm" pod "372f17d9-af5d-4e0c-9ac5-25ee9daffa81" (UID: "372f17d9-af5d-4e0c-9ac5-25ee9daffa81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.333471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.333426    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:39:20.335654 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.335626    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:39:20.340074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.340051    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"]
Oct 31 16:39:20.366999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.366964    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-crcp6"]
Oct 31 16:39:20.374002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.373970    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm\") pod \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") "
Oct 31 16:39:20.374182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache\") pod \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") "
Oct 31 16:39:20.374182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs\") pod \"3b495c62-d368-4270-9576-f59625676332\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") "
Oct 31 16:39:20.374182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374159    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm\") pod \"3b495c62-d368-4270-9576-f59625676332\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") "
Oct 31 16:39:20.374380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374202    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj\" (UniqueName: \"kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj\") pod \"3b495c62-d368-4270-9576-f59625676332\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") "
Oct 31 16:39:20.374380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374239    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir\") pod \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") "
Oct 31 16:39:20.374380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374269    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache\") pod \"3b495c62-d368-4270-9576-f59625676332\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") "
Oct 31 16:39:20.374380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374367    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home\") pod \"3b495c62-d368-4270-9576-f59625676332\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") "
Oct 31 16:39:20.374635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374493    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home\") pod \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") "
Oct 31 16:39:20.374635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374530    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2\" (UniqueName: \"kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2\") pod \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") "
Oct 31 16:39:20.374635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374581    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs\") pod \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\" (UID: \"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd\") "
Oct 31 16:39:20.374635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374625    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir\") pod \"3b495c62-d368-4270-9576-f59625676332\" (UID: \"3b495c62-d368-4270-9576-f59625676332\") "
Oct 31 16:39:20.374865 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.374804    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache" pod "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" (UID: "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.375344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375319    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375349    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375365    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gvg\" (UniqueName: \"kubernetes.io/projected/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-kube-api-access-79gvg\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375381    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375395    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375408    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375422    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372f17d9-af5d-4e0c-9ac5-25ee9daffa81-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.375899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375720    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir" pod "3b495c62-d368-4270-9576-f59625676332" (UID: "3b495c62-d368-4270-9576-f59625676332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.375899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.375729    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache" pod "3b495c62-d368-4270-9576-f59625676332" (UID: "3b495c62-d368-4270-9576-f59625676332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.376022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.376007    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir" pod "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" (UID: "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.376355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.376328    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home" pod "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" (UID: "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.376447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.376405    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home" pod "3b495c62-d368-4270-9576-f59625676332" (UID: "3b495c62-d368-4270-9576-f59625676332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.378791 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.378404    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2" pod "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" (UID: "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd"). InnerVolumeSpecName "kube-api-access-fkhs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:20.380698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.380645    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm" pod "3b495c62-d368-4270-9576-f59625676332" (UID: "3b495c62-d368-4270-9576-f59625676332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.380790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.380724    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs" pod "3b495c62-d368-4270-9576-f59625676332" (UID: "3b495c62-d368-4270-9576-f59625676332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:20.380790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.380746    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj" pod "3b495c62-d368-4270-9576-f59625676332" (UID: "3b495c62-d368-4270-9576-f59625676332"). InnerVolumeSpecName "kube-api-access-6rkhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:20.381001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.380983    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm" pod "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" (UID: "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:20.382956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.382777    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs" pod "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" (UID: "a7364ac2-fb65-43b4-87a6-1bca2d5b28cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:20.440528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.440496    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" path="/var/lib/kubelet/pods/4b8ee24f-7d6e-445f-88f4-08bbf07da809/volumes"
Oct 31 16:39:20.471286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.471255    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"]
Oct 31 16:39:20.476566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476540    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476566 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476566    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b495c62-d368-4270-9576-f59625676332-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476578    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476594    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkhj\" (UniqueName: \"kubernetes.io/projected/3b495c62-d368-4270-9576-f59625676332-kube-api-access-6rkhj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476608    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476620    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476632    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476644    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476657    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhs2\" (UniqueName: \"kubernetes.io/projected/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-kube-api-access-fkhs2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476669    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.476752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476682    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b495c62-d368-4270-9576-f59625676332-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.477201 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:20.476951    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-jkb4n"]
Oct 31 16:39:21.208692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.208597    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"
Oct 31 16:39:21.208692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.208606    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44" event={"ID":"3b495c62-d368-4270-9576-f59625676332","Type":"ContainerDied","Data":"0d385575160d2812b7cf24ffdaa8e4907648223a9598f207bf4ce010c57a8c5c"}
Oct 31 16:39:21.208692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.208656    4519 scope.go:122] "RemoveContainer" containerID="69fe5be07cfe39afbf3bd1d3dc88f4d8b0f43f83e5cd8023c8b5a5cab7fa9ab4"
Oct 31 16:39:21.211604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.211545    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"
Oct 31 16:39:21.212032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.211580    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf" event={"ID":"372f17d9-af5d-4e0c-9ac5-25ee9daffa81","Type":"ContainerDied","Data":"6a891654541b2be426d95755df654af4f9eaad78830b8856d84a74cca70ded3c"}
Oct 31 16:39:21.214675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.214655    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr" event={"ID":"a7364ac2-fb65-43b4-87a6-1bca2d5b28cd","Type":"ContainerDied","Data":"8189105e135fc0ae3264829e0d1d64f2555abc582b8f3386dad27af52fd08d6d"}
Oct 31 16:39:21.214901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.214883    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"
Oct 31 16:39:21.223576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.223557    4519 scope.go:122] "RemoveContainer" containerID="b3b5efc5391874f7566c1359ddfb620d96eac7c6a060a7cda9feb4563c37e406"
Oct 31 16:39:21.234957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.234935    4519 scope.go:122] "RemoveContainer" containerID="274281c410757ec16eaa5392fdb2ddb8cb06f8123dbdcba5d576d047f71c0a3e"
Oct 31 16:39:21.364970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.364939    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"]
Oct 31 16:39:21.368484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.368440    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-2-w2-e7f13a-kserve-567df487b9-mkhmf"]
Oct 31 16:39:21.471678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.471647    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"]
Oct 31 16:39:21.475143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.475114    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/distinct-w2-e7f13a-kserve-584c5fb47d-7dgpr"]
Oct 31 16:39:21.575811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.575777    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"]
Oct 31 16:39:21.579068 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:21.579043    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/premium-w2-e7f13a-kserve-65fd447687-7gn44"]
Oct 31 16:39:22.443505 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:22.443469    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" path="/var/lib/kubelet/pods/372f17d9-af5d-4e0c-9ac5-25ee9daffa81/volumes"
Oct 31 16:39:22.443954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:22.443939    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b495c62-d368-4270-9576-f59625676332" path="/var/lib/kubelet/pods/3b495c62-d368-4270-9576-f59625676332/volumes"
Oct 31 16:39:22.444336 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:22.444324    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" path="/var/lib/kubelet/pods/a7364ac2-fb65-43b4-87a6-1bca2d5b28cd/volumes"
Oct 31 16:39:22.445078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:22.445062    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" path="/var/lib/kubelet/pods/bc3c224c-3bdd-410a-b0bc-4ff880c613a3/volumes"
Oct 31 16:39:23.290914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:23.290873    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:39:23.291089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:23.290927    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:39:24.006435 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:24.006391    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:39:24.562181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:24.562142    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"]
Oct 31 16:39:24.609219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:24.609181    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3dc1da-tmtqd"]
Oct 31 16:39:26.439648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:26.439617    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe0e699-4034-4877-98e1-2b7e528bbb41" path="/var/lib/kubelet/pods/bbe0e699-4034-4877-98e1-2b7e528bbb41/volumes"
Oct 31 16:39:26.492563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:26.492524    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"]
Oct 31 16:39:26.492794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:26.492770    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy" containerID="cri-o://a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87" gracePeriod=30
Oct 31 16:39:31.270118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.270082    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"]
Oct 31 16:39:31.270656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.270374    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy" containerID="cri-o://f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4" gracePeriod=30
Oct 31 16:39:31.797643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.797573    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:39:31.952587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952537    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.952785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952611    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.952785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952650    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.952785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952677    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.952785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952730    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.952785 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952760    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.953135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952787    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.953135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952862    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.953135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.952913    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data\") pod \"0de72231-0262-40b5-87b4-8a2154737fa7\" (UID: \"0de72231-0262-40b5-87b4-8a2154737fa7\") "
Oct 31 16:39:31.953327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.953296    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:31.953536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.953515    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de72231-0262-40b5-87b4-8a2154737fa7-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:31.953636 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.953612    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:31.953965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.953937    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:31.954250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.954226    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:31.954382 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.954359    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:31.957245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.956681    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:31.957245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.956747    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "kube-api-access-f9bbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:31.957245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.957112    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:31.957245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:31.957202    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo" pod "0de72231-0262-40b5-87b4-8a2154737fa7" (UID: "0de72231-0262-40b5-87b4-8a2154737fa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054767    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054797    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054807    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bbt\" (UniqueName: \"kubernetes.io/projected/0de72231-0262-40b5-87b4-8a2154737fa7-kube-api-access-f9bbt\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054817    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054827    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054836    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054844    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de72231-0262-40b5-87b4-8a2154737fa7-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.054849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.054852    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de72231-0262-40b5-87b4-8a2154737fa7-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:32.299378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.299339    4519 generic.go:356] "Generic (PLEG): container finished" podID="0de72231-0262-40b5-87b4-8a2154737fa7" containerID="a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87" exitCode=0
Oct 31 16:39:32.299822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.299403    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"
Oct 31 16:39:32.299822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.299419    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" event={"ID":"0de72231-0262-40b5-87b4-8a2154737fa7","Type":"ContainerDied","Data":"a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87"}
Oct 31 16:39:32.299822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.299510    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk" event={"ID":"0de72231-0262-40b5-87b4-8a2154737fa7","Type":"ContainerDied","Data":"7cda9c63c60a2c8ee2d93aefa4ff4c5f0d4b0380ad7a5b6d2e71104c9b1ab2cc"}
Oct 31 16:39:32.299822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.299538    4519 scope.go:122] "RemoveContainer" containerID="a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87"
Oct 31 16:39:32.311243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.311225    4519 scope.go:122] "RemoveContainer" containerID="49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"
Oct 31 16:39:32.321202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.321186    4519 scope.go:122] "RemoveContainer" containerID="a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87"
Oct 31 16:39:32.321504 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:32.321483    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87\": container with ID starting with a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87 not found: ID does not exist" containerID="a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87"
Oct 31 16:39:32.321579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.321513    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87"} err="failed to get container status \"a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87\": rpc error: code = NotFound desc = could not find container \"a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87\": container with ID starting with a70bf32781221df9e21813a35685f24f19aae676e4024f2c6a9d431cc9ab2b87 not found: ID does not exist"
Oct 31 16:39:32.321579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.321532    4519 scope.go:122] "RemoveContainer" containerID="49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"
Oct 31 16:39:32.321740 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:32.321727    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5\": container with ID starting with 49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5 not found: ID does not exist" containerID="49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"
Oct 31 16:39:32.321788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.321744    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5"} err="failed to get container status \"49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5\": rpc error: code = NotFound desc = could not find container \"49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5\": container with ID starting with 49a805992e0ed94ba76a167f0b58b3af5a319da6ad2f964c2f6db42e6d009fd5 not found: ID does not exist"
Oct 31 16:39:32.331873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.331849    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"]
Oct 31 16:39:32.332522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332498    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main"
Oct 31 16:39:32.332522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332520    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332535    4519 cpu_manager.go:414] "containerMap: removing container" podUID="125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" containerName="revoke-keys"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332541    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" containerName="revoke-keys"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332549    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332554    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332574    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332579    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332605    4519 cpu_manager.go:414] "containerMap: removing container" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332610    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332620    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332624    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332644    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy"
Oct 31 16:39:32.332670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332648    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332741    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332751    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8ee24f-7d6e-445f-88f4-08bbf07da809" containerName="istio-proxy"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332761    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="372f17d9-af5d-4e0c-9ac5-25ee9daffa81" containerName="main"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332768    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7364ac2-fb65-43b4-87a6-1bca2d5b28cd" containerName="main"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332775    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3c224c-3bdd-410a-b0bc-4ff880c613a3" containerName="istio-proxy"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332782    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b495c62-d368-4270-9576-f59625676332" containerName="main"
Oct 31 16:39:32.333027 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.332786    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" containerName="revoke-keys"
Oct 31 16:39:32.337549 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.337533    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.388632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.387313    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:39:32.388632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.387349    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:39:32.388632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.387369    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:39:32.388632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.387402    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:39:32.388632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.388577    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"]
Oct 31 16:39:32.459587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.459555    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.459792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.459772    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7pf\" (UniqueName: \"kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.459846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.459833    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.561165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.561083    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7pf\" (UniqueName: \"kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.561165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.561143    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.561391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.561212    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.562094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.562056    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.563855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.563828    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.568556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.568529    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"]
Oct 31 16:39:32.573779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.573748    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-wzcpk"]
Oct 31 16:39:32.615102 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.615072    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7pf\" (UniqueName: \"kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf\") pod \"maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:32.651293 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:32.651270    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:33.023054 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:39:33.023019    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9e5dc2_f15d_4ba7_85a2_fdace7537c1f.slice/crio-4ebef45acfb6537c8590af423906a8067261450b1d7f2209941a6de7f04db7e2 WatchSource:0}: Error finding container 4ebef45acfb6537c8590af423906a8067261450b1d7f2209941a6de7f04db7e2: Status 404 returned error can't find the container with id 4ebef45acfb6537c8590af423906a8067261450b1d7f2209941a6de7f04db7e2
Oct 31 16:39:33.023618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:33.023582    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"]
Oct 31 16:39:33.290908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:33.290871    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:39:33.290908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:33.290896    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:39:33.309109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:33.309070    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw" event={"ID":"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f","Type":"ContainerStarted","Data":"43fc24387ee02fec6bf97f41e7d6cf594f60d1cbf0bcbba5e90bbac27c370fd2"}
Oct 31 16:39:33.309585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:33.309138    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw" event={"ID":"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f","Type":"ContainerStarted","Data":"4ebef45acfb6537c8590af423906a8067261450b1d7f2209941a6de7f04db7e2"}
Oct 31 16:39:33.417149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:33.417092    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw" podStartSLOduration=1.417077039 podStartE2EDuration="1.417077039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:39:33.416649982 +0000 UTC m=+2217.116579293" watchObservedRunningTime="2026-10-31 16:39:33.417077039 +0000 UTC m=+2217.117006329"
Oct 31 16:39:34.319335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:34.319294    4519 generic.go:356] "Generic (PLEG): container finished" podID="cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" containerID="43fc24387ee02fec6bf97f41e7d6cf594f60d1cbf0bcbba5e90bbac27c370fd2" exitCode=0
Oct 31 16:39:34.319830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:34.319384    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw" event={"ID":"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f","Type":"ContainerDied","Data":"43fc24387ee02fec6bf97f41e7d6cf594f60d1cbf0bcbba5e90bbac27c370fd2"}
Oct 31 16:39:34.440649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:34.440617    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" path="/var/lib/kubelet/pods/0de72231-0262-40b5-87b4-8a2154737fa7/volumes"
Oct 31 16:39:35.500837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.500809    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:35.602737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.602660    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca\") pod \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") "
Oct 31 16:39:35.602903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.602794    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf\" (UniqueName: \"kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf\") pod \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") "
Oct 31 16:39:35.602903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.602843    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca\") pod \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\" (UID: \"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f\") "
Oct 31 16:39:35.603064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.603036    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca" pod "cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" (UID: "cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:35.603285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.603259    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:35.604864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.604845    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf" pod "cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" (UID: "cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f"). InnerVolumeSpecName "kube-api-access-5z7pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:35.604864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.604852    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca" pod "cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" (UID: "cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:35.704766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.704732    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7pf\" (UniqueName: \"kubernetes.io/projected/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-kube-api-access-5z7pf\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:35.704766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:35.704760    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.335420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.335394    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"
Oct 31 16:39:36.335631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.335425    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw" event={"ID":"cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f","Type":"ContainerDied","Data":"4ebef45acfb6537c8590af423906a8067261450b1d7f2209941a6de7f04db7e2"}
Oct 31 16:39:36.335631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.335481    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ebef45acfb6537c8590af423906a8067261450b1d7f2209941a6de7f04db7e2"
Oct 31 16:39:36.573958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.573930    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:39:36.730294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730259    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730340    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730393    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730422    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730491    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730533    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730723    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730723    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:36.730882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730788    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.730882 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.730822    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket\") pod \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\" (UID: \"9393cbea-c7d6-47d7-9a20-a52a318ef5e6\") "
Oct 31 16:39:36.731340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.731318    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.732254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.731737    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:36.732515 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.731869    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:36.732688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.731958    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:36.732812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.732074    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:36.733049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.733029    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:36.733646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.733616    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:36.734086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.734062    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "kube-api-access-z7tj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:36.734537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.734512    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy" pod "9393cbea-c7d6-47d7-9a20-a52a318ef5e6" (UID: "9393cbea-c7d6-47d7-9a20-a52a318ef5e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:36.794519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.794391    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"]
Oct 31 16:39:36.833809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833782    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833808    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833821    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833832    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833841    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833851    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833860    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7tj9\" (UniqueName: \"kubernetes.io/projected/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-kube-api-access-z7tj9\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:36.833963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:36.833869    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9393cbea-c7d6-47d7-9a20-a52a318ef5e6-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:37.054961 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.054924    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:39:37.189999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.189976    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:39:37.238973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.238938    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume\") pod \"6ee7f68b-16e6-4338-b2ed-934506f988b4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") "
Oct 31 16:39:37.239134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.239001    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh\" (UniqueName: \"kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh\") pod \"6ee7f68b-16e6-4338-b2ed-934506f988b4\" (UID: \"6ee7f68b-16e6-4338-b2ed-934506f988b4\") "
Oct 31 16:39:37.239383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.239359    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume" pod "6ee7f68b-16e6-4338-b2ed-934506f988b4" (UID: "6ee7f68b-16e6-4338-b2ed-934506f988b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:37.241148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.241122    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh" pod "6ee7f68b-16e6-4338-b2ed-934506f988b4" (UID: "6ee7f68b-16e6-4338-b2ed-934506f988b4"). InnerVolumeSpecName "kube-api-access-jqrxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:37.340260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.340179    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6\" (UniqueName: \"kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6\") pod \"013c6457-ed0f-4a16-959a-8bec9e9cec92\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") "
Oct 31 16:39:37.340860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.340440    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume\") pod \"013c6457-ed0f-4a16-959a-8bec9e9cec92\" (UID: \"013c6457-ed0f-4a16-959a-8bec9e9cec92\") "
Oct 31 16:39:37.340860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.340829    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume" pod "013c6457-ed0f-4a16-959a-8bec9e9cec92" (UID: "013c6457-ed0f-4a16-959a-8bec9e9cec92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:37.341046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.341023    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrxh\" (UniqueName: \"kubernetes.io/projected/6ee7f68b-16e6-4338-b2ed-934506f988b4-kube-api-access-jqrxh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:37.341046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.341042    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee7f68b-16e6-4338-b2ed-934506f988b4-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:37.341150 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.341054    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/013c6457-ed0f-4a16-959a-8bec9e9cec92-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:37.342976 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.342951    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6" pod "013c6457-ed0f-4a16-959a-8bec9e9cec92" (UID: "013c6457-ed0f-4a16-959a-8bec9e9cec92"). InnerVolumeSpecName "kube-api-access-lf2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:37.346213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.346189    4519 generic.go:356] "Generic (PLEG): container finished" podID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerID="de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27" exitCode=0
Oct 31 16:39:37.346310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.346236    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"
Oct 31 16:39:37.346310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.346274    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" event={"ID":"6ee7f68b-16e6-4338-b2ed-934506f988b4","Type":"ContainerDied","Data":"de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27"}
Oct 31 16:39:37.346430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.346355    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4" event={"ID":"6ee7f68b-16e6-4338-b2ed-934506f988b4","Type":"ContainerDied","Data":"96369b66e498dc4fc6e2deab8a937272048477c023fc78f4f63fd5c766f204f4"}
Oct 31 16:39:37.346430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.346383    4519 scope.go:122] "RemoveContainer" containerID="de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27"
Oct 31 16:39:37.348033 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.348008    4519 generic.go:356] "Generic (PLEG): container finished" podID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerID="f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4" exitCode=0
Oct 31 16:39:37.348138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.348065    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"
Oct 31 16:39:37.348138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.348085    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" event={"ID":"9393cbea-c7d6-47d7-9a20-a52a318ef5e6","Type":"ContainerDied","Data":"f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4"}
Oct 31 16:39:37.348138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.348129    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f" event={"ID":"9393cbea-c7d6-47d7-9a20-a52a318ef5e6","Type":"ContainerDied","Data":"aec57ce937c9df4190fc203d32da08bd674921038f2ab7a47cda09c3d10deff1"}
Oct 31 16:39:37.349973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.349948    4519 generic.go:356] "Generic (PLEG): container finished" podID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerID="764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c" exitCode=0
Oct 31 16:39:37.350167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.350005    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"
Oct 31 16:39:37.350301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.350021    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" event={"ID":"013c6457-ed0f-4a16-959a-8bec9e9cec92","Type":"ContainerDied","Data":"764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c"}
Oct 31 16:39:37.350301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.350279    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9" event={"ID":"013c6457-ed0f-4a16-959a-8bec9e9cec92","Type":"ContainerDied","Data":"1614c1f1cbfb8b77c26bd6abd0bc885e10dd57c23bd72d50949805b2b457c008"}
Oct 31 16:39:37.357707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.357688    4519 scope.go:122] "RemoveContainer" containerID="de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27"
Oct 31 16:39:37.358008 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:37.357985    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27\": container with ID starting with de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27 not found: ID does not exist" containerID="de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27"
Oct 31 16:39:37.358104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.358015    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27"} err="failed to get container status \"de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27\": rpc error: code = NotFound desc = could not find container \"de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27\": container with ID starting with de23d08fb5d9f6231d21382e54fd9dda082f322f80adc272bd78583f2ba0ce27 not found: ID does not exist"
Oct 31 16:39:37.358104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.358046    4519 scope.go:122] "RemoveContainer" containerID="f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4"
Oct 31 16:39:37.367819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.367801    4519 scope.go:122] "RemoveContainer" containerID="9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"
Oct 31 16:39:37.377276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.377259    4519 scope.go:122] "RemoveContainer" containerID="f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4"
Oct 31 16:39:37.377564 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:37.377542    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4\": container with ID starting with f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4 not found: ID does not exist" containerID="f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4"
Oct 31 16:39:37.377618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.377571    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4"} err="failed to get container status \"f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4\": rpc error: code = NotFound desc = could not find container \"f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4\": container with ID starting with f101cc1c8890f7c6cdcc0518d6113da537da0126b3d69efeb8e5cdf3b41783c4 not found: ID does not exist"
Oct 31 16:39:37.377618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.377589    4519 scope.go:122] "RemoveContainer" containerID="9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"
Oct 31 16:39:37.377822 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:37.377802    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f\": container with ID starting with 9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f not found: ID does not exist" containerID="9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"
Oct 31 16:39:37.377899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.377832    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f"} err="failed to get container status \"9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f\": rpc error: code = NotFound desc = could not find container \"9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f\": container with ID starting with 9f632d86933ec6515acff1b7ae561a02154102a336997157060f8e9cd508c99f not found: ID does not exist"
Oct 31 16:39:37.377899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.377856    4519 scope.go:122] "RemoveContainer" containerID="764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c"
Oct 31 16:39:37.386935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.386915    4519 scope.go:122] "RemoveContainer" containerID="764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c"
Oct 31 16:39:37.387207 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:37.387186    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c\": container with ID starting with 764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c not found: ID does not exist" containerID="764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c"
Oct 31 16:39:37.387273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.387220    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c"} err="failed to get container status \"764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c\": rpc error: code = NotFound desc = could not find container \"764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c\": container with ID starting with 764f0473a718bb2e318187f94c378c7e48369e320d3c00b4c91313bb6d86798c not found: ID does not exist"
Oct 31 16:39:37.441872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.441832    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2r6\" (UniqueName: \"kubernetes.io/projected/013c6457-ed0f-4a16-959a-8bec9e9cec92-kube-api-access-lf2r6\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:37.503518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.503477    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"]
Oct 31 16:39:37.506723 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.506694    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3dc1da-openshift-default-64679f6975-6lz7f"]
Oct 31 16:39:37.608915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.608840    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"]
Oct 31 16:39:37.616149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.616120    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7f13a-687d7c9f45-wt5d4"]
Oct 31 16:39:37.718166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.718125    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"]
Oct 31 16:39:37.721231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:37.721208    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7f13a-fff8d7787-f82q9"]
Oct 31 16:39:38.439465 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:38.439430    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" path="/var/lib/kubelet/pods/013c6457-ed0f-4a16-959a-8bec9e9cec92/volumes"
Oct 31 16:39:38.439943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:38.439929    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" path="/var/lib/kubelet/pods/6ee7f68b-16e6-4338-b2ed-934506f988b4/volumes"
Oct 31 16:39:38.440373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:38.440360    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" path="/var/lib/kubelet/pods/9393cbea-c7d6-47d7-9a20-a52a318ef5e6/volumes"
Oct 31 16:39:38.622866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:38.622831    4519 scope.go:122] "RemoveContainer" containerID="29d1cc44b94f8a27832ae6ba253cff0587d46df67695d5445ce4b91870092464"
Oct 31 16:39:39.394895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.394862    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"]
Oct 31 16:39:39.395504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395490    4519 cpu_manager.go:414] "containerMap: removing container" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerName="payload-pre-processing"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395506    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerName="payload-pre-processing"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395516    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" containerName="revoke-keys"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395521    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" containerName="revoke-keys"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395541    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395546    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395558    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395562    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395571    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerName="payload-processing"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395576    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerName="payload-processing"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395583    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy"
Oct 31 16:39:39.395588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395587    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy"
Oct 31 16:39:39.395915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395671    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="013c6457-ed0f-4a16-959a-8bec9e9cec92" containerName="payload-pre-processing"
Oct 31 16:39:39.395915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395679    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" containerName="revoke-keys"
Oct 31 16:39:39.395915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395690    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de72231-0262-40b5-87b4-8a2154737fa7" containerName="istio-proxy"
Oct 31 16:39:39.395915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395697    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy"
Oct 31 16:39:39.395915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395703    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee7f68b-16e6-4338-b2ed-934506f988b4" containerName="payload-processing"
Oct 31 16:39:39.395915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.395710    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9393cbea-c7d6-47d7-9a20-a52a318ef5e6" containerName="istio-proxy"
Oct 31 16:39:39.401661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.401639    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.452334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.452303    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"]
Oct 31 16:39:39.467558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.467530    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.467858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.467837    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.468176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.468154    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2fs\" (UniqueName: \"kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.494701 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.494664    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:39:39.494701 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.494679    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:39:39.494922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.494736    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:39:39.498716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.498692    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:39:39.506752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.505104    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"]
Oct 31 16:39:39.513580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.513547    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.549470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.549422    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"]
Oct 31 16:39:39.569188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.569154    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5tq\" (UniqueName: \"kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.569362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.569196    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.569408 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.569369    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.569482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.569465    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.569587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.569572    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2fs\" (UniqueName: \"kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.569667 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.569652    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.570187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.570160    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.572001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.571970    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.623519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.623486    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2fs\" (UniqueName: \"kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs\") pod \"maas-api-revoke-sub-c009319b-fnwj6\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.671820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.671710    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5tq\" (UniqueName: \"kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.671820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.671765    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.671820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.671810    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.672576 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.672504    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.674649 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.674624    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.720772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.720741    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:39.728368 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.728341    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5tq\" (UniqueName: \"kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq\") pod \"maas-api-revoke-sub-d97adbbd-qnl4q\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:39.831060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:39.830785    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:40.131111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:40.131077    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"]
Oct 31 16:39:40.132822 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:39:40.132790    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec88070_714e_419e_a0ad_0799a40c9cac.slice/crio-e57c1558113c217e90e4fb58ce80c0039c930b84c8efaf393b8ce69e71eb3a8f WatchSource:0}: Error finding container e57c1558113c217e90e4fb58ce80c0039c930b84c8efaf393b8ce69e71eb3a8f: Status 404 returned error can't find the container with id e57c1558113c217e90e4fb58ce80c0039c930b84c8efaf393b8ce69e71eb3a8f
Oct 31 16:39:40.225348 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:40.225319    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"]
Oct 31 16:39:40.225348 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:39:40.225333    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3197fc09_35f8_44fe_80b1_e818f2479d06.slice/crio-fd076e79e2bd6f32faab98a3ec6cf850b740590dd14b6d98a1a2ae8db4bf3226 WatchSource:0}: Error finding container fd076e79e2bd6f32faab98a3ec6cf850b740590dd14b6d98a1a2ae8db4bf3226: Status 404 returned error can't find the container with id fd076e79e2bd6f32faab98a3ec6cf850b740590dd14b6d98a1a2ae8db4bf3226
Oct 31 16:39:40.377364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:40.377327    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6" event={"ID":"1ec88070-714e-419e-a0ad-0799a40c9cac","Type":"ContainerStarted","Data":"e57c1558113c217e90e4fb58ce80c0039c930b84c8efaf393b8ce69e71eb3a8f"}
Oct 31 16:39:40.378629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:40.378570    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q" event={"ID":"3197fc09-35f8-44fe-80b1-e818f2479d06","Type":"ContainerStarted","Data":"fd076e79e2bd6f32faab98a3ec6cf850b740590dd14b6d98a1a2ae8db4bf3226"}
Oct 31 16:39:41.388016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:41.387981    4519 generic.go:356] "Generic (PLEG): container finished" podID="1ec88070-714e-419e-a0ad-0799a40c9cac" containerID="92ecfc718ef2847050a5d2d5353f5b5f9925dcc7436a0bd882c3f542b0c3f979" exitCode=0
Oct 31 16:39:41.388504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:41.388062    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6" event={"ID":"1ec88070-714e-419e-a0ad-0799a40c9cac","Type":"ContainerDied","Data":"92ecfc718ef2847050a5d2d5353f5b5f9925dcc7436a0bd882c3f542b0c3f979"}
Oct 31 16:39:41.389773 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:41.389741    4519 generic.go:356] "Generic (PLEG): container finished" podID="3197fc09-35f8-44fe-80b1-e818f2479d06" containerID="831b0f96644acfab7a7830ced5924475b980de44d03d41040e87955124e3ae28" exitCode=0
Oct 31 16:39:41.389858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:41.389796    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q" event={"ID":"3197fc09-35f8-44fe-80b1-e818f2479d06","Type":"ContainerDied","Data":"831b0f96644acfab7a7830ced5924475b980de44d03d41040e87955124e3ae28"}
Oct 31 16:39:41.937425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:41.937388    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"]
Oct 31 16:39:41.940868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:41.940814    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1b30a15-5mfrj"]
Oct 31 16:39:42.441350 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.441311    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1f2769-2825-43fd-9324-53992558b371" path="/var/lib/kubelet/pods/3c1f2769-2825-43fd-9324-53992558b371/volumes"
Oct 31 16:39:42.639894 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.639864    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:42.642437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.642395    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:42.765335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765301    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca\") pod \"3197fc09-35f8-44fe-80b1-e818f2479d06\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") "
Oct 31 16:39:42.765597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765369    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca\") pod \"1ec88070-714e-419e-a0ad-0799a40c9cac\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") "
Oct 31 16:39:42.765597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765399    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca\") pod \"3197fc09-35f8-44fe-80b1-e818f2479d06\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") "
Oct 31 16:39:42.765597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765577    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs\" (UniqueName: \"kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs\") pod \"1ec88070-714e-419e-a0ad-0799a40c9cac\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") "
Oct 31 16:39:42.765802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765682    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq\" (UniqueName: \"kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq\") pod \"3197fc09-35f8-44fe-80b1-e818f2479d06\" (UID: \"3197fc09-35f8-44fe-80b1-e818f2479d06\") "
Oct 31 16:39:42.765802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765722    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca\") pod \"1ec88070-714e-419e-a0ad-0799a40c9cac\" (UID: \"1ec88070-714e-419e-a0ad-0799a40c9cac\") "
Oct 31 16:39:42.765926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.765889    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca" pod "3197fc09-35f8-44fe-80b1-e818f2479d06" (UID: "3197fc09-35f8-44fe-80b1-e818f2479d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:42.766581 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.766550    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:42.767342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.767287    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca" pod "1ec88070-714e-419e-a0ad-0799a40c9cac" (UID: "1ec88070-714e-419e-a0ad-0799a40c9cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:42.768304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.768279    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca" pod "3197fc09-35f8-44fe-80b1-e818f2479d06" (UID: "3197fc09-35f8-44fe-80b1-e818f2479d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:42.768546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.768520    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca" pod "1ec88070-714e-419e-a0ad-0799a40c9cac" (UID: "1ec88070-714e-419e-a0ad-0799a40c9cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:42.769406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.769385    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs" pod "1ec88070-714e-419e-a0ad-0799a40c9cac" (UID: "1ec88070-714e-419e-a0ad-0799a40c9cac"). InnerVolumeSpecName "kube-api-access-dd2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:42.770113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.770095    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq" pod "3197fc09-35f8-44fe-80b1-e818f2479d06" (UID: "3197fc09-35f8-44fe-80b1-e818f2479d06"). InnerVolumeSpecName "kube-api-access-dt5tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:42.868098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.868060    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:42.868098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.868101    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3197fc09-35f8-44fe-80b1-e818f2479d06-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:42.868282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.868118    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2fs\" (UniqueName: \"kubernetes.io/projected/1ec88070-714e-419e-a0ad-0799a40c9cac-kube-api-access-dd2fs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:42.868282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.868134    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5tq\" (UniqueName: \"kubernetes.io/projected/3197fc09-35f8-44fe-80b1-e818f2479d06-kube-api-access-dt5tq\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:42.868282 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:42.868150    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec88070-714e-419e-a0ad-0799a40c9cac-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:43.193379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.193289    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"]
Oct 31 16:39:43.297099 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.297061    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"]
Oct 31 16:39:43.418213 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.418184    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"
Oct 31 16:39:43.418404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.418219    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6" event={"ID":"1ec88070-714e-419e-a0ad-0799a40c9cac","Type":"ContainerDied","Data":"e57c1558113c217e90e4fb58ce80c0039c930b84c8efaf393b8ce69e71eb3a8f"}
Oct 31 16:39:43.418404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.418278    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e57c1558113c217e90e4fb58ce80c0039c930b84c8efaf393b8ce69e71eb3a8f"
Oct 31 16:39:43.419620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.419591    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q" event={"ID":"3197fc09-35f8-44fe-80b1-e818f2479d06","Type":"ContainerDied","Data":"fd076e79e2bd6f32faab98a3ec6cf850b740590dd14b6d98a1a2ae8db4bf3226"}
Oct 31 16:39:43.419620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.419616    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"
Oct 31 16:39:43.419764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.419629    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd076e79e2bd6f32faab98a3ec6cf850b740590dd14b6d98a1a2ae8db4bf3226"
Oct 31 16:39:43.825230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825188    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"]
Oct 31 16:39:43.825825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825808    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1ec88070-714e-419e-a0ad-0799a40c9cac" containerName="revoke-keys"
Oct 31 16:39:43.825871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825826    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec88070-714e-419e-a0ad-0799a40c9cac" containerName="revoke-keys"
Oct 31 16:39:43.825908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825873    4519 cpu_manager.go:414] "containerMap: removing container" podUID="3197fc09-35f8-44fe-80b1-e818f2479d06" containerName="revoke-keys"
Oct 31 16:39:43.825908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825879    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3197fc09-35f8-44fe-80b1-e818f2479d06" containerName="revoke-keys"
Oct 31 16:39:43.825969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825963    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="3197fc09-35f8-44fe-80b1-e818f2479d06" containerName="revoke-keys"
Oct 31 16:39:43.826003 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.825972    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec88070-714e-419e-a0ad-0799a40c9cac" containerName="revoke-keys"
Oct 31 16:39:43.829772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.829752    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.881707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.881671    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.881902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.881763    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.881902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.881804    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.881902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.881858    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghlcz\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.882118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.881952    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.882118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.881980    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.882118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.882006    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.882118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.882088    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.882333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.882163    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.886039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.885997    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"]
Oct 31 16:39:43.983696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.983659    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.983885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.983713    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghlcz\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.983885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.983777    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.983891    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.983940    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984024    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984053    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984109    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984114    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984194    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984210    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984274    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984289    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.984729 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.984711    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.986383 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.986354    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:43.986541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:43.986524    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:44.038217 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.038181    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:44.038374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.038358    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghlcz\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cpsqt\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:44.144441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.144357    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:44.561734 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:39:44.561699    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf17d887_3bc7_4077_a2dc_755c296a3b12.slice/crio-f6a62afb6bfbe2e713a94fd6e0de048690eadfd4eed478f622889f5a7b6ce6f6 WatchSource:0}: Error finding container f6a62afb6bfbe2e713a94fd6e0de048690eadfd4eed478f622889f5a7b6ce6f6: Status 404 returned error can't find the container with id f6a62afb6bfbe2e713a94fd6e0de048690eadfd4eed478f622889f5a7b6ce6f6
Oct 31 16:39:44.562235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.562207    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"]
Oct 31 16:39:44.564668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.564629    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:44.564768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.564700    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:44.564768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:44.564728    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:45.434096 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:45.434061    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" event={"ID":"af17d887-3bc7-4077-a2dc-755c296a3b12","Type":"ContainerStarted","Data":"8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373"}
Oct 31 16:39:45.434519 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:45.434112    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" event={"ID":"af17d887-3bc7-4077-a2dc-755c296a3b12","Type":"ContainerStarted","Data":"f6a62afb6bfbe2e713a94fd6e0de048690eadfd4eed478f622889f5a7b6ce6f6"}
Oct 31 16:39:45.643532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:45.643474    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podStartSLOduration=2.643450041 podStartE2EDuration="2.643450041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:39:45.59419935 +0000 UTC m=+2229.294128662" watchObservedRunningTime="2026-10-31 16:39:45.643450041 +0000 UTC m=+2229.343379352"
Oct 31 16:39:46.144633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:46.144593    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:46.146542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:46.146515    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:46.146656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:46.146564    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:47.145602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:47.145568    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:47.145995 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:47.145622    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:48.145044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.145006    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:48.145253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.145083    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:48.193813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.193754    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh" podUID="ba368270-cdbc-419e-85cb-74a6e8a0fab7" containerName="payload-processing" containerID="cri-o://ca6db2d2ac7503fe1740ad1b1e33d0eb7ae6af3cad61202546cc135e628bf330" gracePeriod=25
Oct 31 16:39:48.297482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.297419    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" podUID="ce91a06d-620e-4c7f-9317-313db927a913" containerName="payload-pre-processing" containerID="cri-o://6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37" gracePeriod=25
Oct 31 16:39:48.461795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.461760    4519 generic.go:356] "Generic (PLEG): container finished" podID="ba368270-cdbc-419e-85cb-74a6e8a0fab7" containerID="ca6db2d2ac7503fe1740ad1b1e33d0eb7ae6af3cad61202546cc135e628bf330" exitCode=0
Oct 31 16:39:48.461973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.461799    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh" event={"ID":"ba368270-cdbc-419e-85cb-74a6e8a0fab7","Type":"ContainerDied","Data":"ca6db2d2ac7503fe1740ad1b1e33d0eb7ae6af3cad61202546cc135e628bf330"}
Oct 31 16:39:48.517687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.517413    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:39:48.657871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.657837    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx\" (UniqueName: \"kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx\") pod \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") "
Oct 31 16:39:48.658057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.657924    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume\") pod \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\" (UID: \"ba368270-cdbc-419e-85cb-74a6e8a0fab7\") "
Oct 31 16:39:48.658391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.658358    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume" pod "ba368270-cdbc-419e-85cb-74a6e8a0fab7" (UID: "ba368270-cdbc-419e-85cb-74a6e8a0fab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:48.658711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.658691    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba368270-cdbc-419e-85cb-74a6e8a0fab7-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:48.661842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.661811    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx" pod "ba368270-cdbc-419e-85cb-74a6e8a0fab7" (UID: "ba368270-cdbc-419e-85cb-74a6e8a0fab7"). InnerVolumeSpecName "kube-api-access-s7vfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:48.760264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:48.760174    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7vfx\" (UniqueName: \"kubernetes.io/projected/ba368270-cdbc-419e-85cb-74a6e8a0fab7-kube-api-access-s7vfx\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:49.145769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.145675    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:49.145769 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.145752    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:49.491231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.491196    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"
Oct 31 16:39:49.491231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.491219    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh" event={"ID":"ba368270-cdbc-419e-85cb-74a6e8a0fab7","Type":"ContainerDied","Data":"838a73725c962c7768c37320248cab9d25aa33e7259b1100090264088ceef530"}
Oct 31 16:39:49.491852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.491296    4519 scope.go:122] "RemoveContainer" containerID="ca6db2d2ac7503fe1740ad1b1e33d0eb7ae6af3cad61202546cc135e628bf330"
Oct 31 16:39:49.649449 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.649407    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"]
Oct 31 16:39:49.698417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:49.698383    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4fe3a4-6474577947-xj4nh"]
Oct 31 16:39:50.144920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:50.144885    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:50.145122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:50.144941    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:50.439678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:50.439598    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba368270-cdbc-419e-85cb-74a6e8a0fab7" path="/var/lib/kubelet/pods/ba368270-cdbc-419e-85cb-74a6e8a0fab7/volumes"
Oct 31 16:39:51.145228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:51.145190    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:51.145780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:51.145251    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:51.816228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:51.816182    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" containerID="cri-o://1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa" gracePeriod=45
Oct 31 16:39:52.089985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.089956    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:39:52.145588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.145555    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cpsqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:39:52.145985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.145627    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:39:52.247251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.246007    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir\") pod \"6761d701-7651-4a9f-b397-0f29df36ce96\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") "
Oct 31 16:39:52.247251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.246134    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs\" (UniqueName: \"kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs\") pod \"6761d701-7651-4a9f-b397-0f29df36ce96\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") "
Oct 31 16:39:52.247251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.246203    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache\") pod \"6761d701-7651-4a9f-b397-0f29df36ce96\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") "
Oct 31 16:39:52.247251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.246239    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm\") pod \"6761d701-7651-4a9f-b397-0f29df36ce96\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") "
Oct 31 16:39:52.247251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.246277    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home\") pod \"6761d701-7651-4a9f-b397-0f29df36ce96\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") "
Oct 31 16:39:52.247251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.246342    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl\" (UniqueName: \"kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl\") pod \"6761d701-7651-4a9f-b397-0f29df36ce96\" (UID: \"6761d701-7651-4a9f-b397-0f29df36ce96\") "
Oct 31 16:39:52.248155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.247989    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache" pod "6761d701-7651-4a9f-b397-0f29df36ce96" (UID: "6761d701-7651-4a9f-b397-0f29df36ce96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.248583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.248292    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir" pod "6761d701-7651-4a9f-b397-0f29df36ce96" (UID: "6761d701-7651-4a9f-b397-0f29df36ce96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.249209 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.249176    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home" pod "6761d701-7651-4a9f-b397-0f29df36ce96" (UID: "6761d701-7651-4a9f-b397-0f29df36ce96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.289260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.289201    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm" pod "6761d701-7651-4a9f-b397-0f29df36ce96" (UID: "6761d701-7651-4a9f-b397-0f29df36ce96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.332034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.331973    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl" pod "6761d701-7651-4a9f-b397-0f29df36ce96" (UID: "6761d701-7651-4a9f-b397-0f29df36ce96"). InnerVolumeSpecName "kube-api-access-fl4wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:52.334513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.334446    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs" pod "6761d701-7651-4a9f-b397-0f29df36ce96" (UID: "6761d701-7651-4a9f-b397-0f29df36ce96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:52.351751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.351614    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4wl\" (UniqueName: \"kubernetes.io/projected/6761d701-7651-4a9f-b397-0f29df36ce96-kube-api-access-fl4wl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.351751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.351651    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.351751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.351669    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6761d701-7651-4a9f-b397-0f29df36ce96-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.351751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.351683    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.351751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.351699    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.351751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.351714    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6761d701-7651-4a9f-b397-0f29df36ce96-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.529476 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.529426    4519 generic.go:356] "Generic (PLEG): container finished" podID="6761d701-7651-4a9f-b397-0f29df36ce96" containerID="1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa" exitCode=0
Oct 31 16:39:52.529648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.529503    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"
Oct 31 16:39:52.529648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.529515    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" event={"ID":"6761d701-7651-4a9f-b397-0f29df36ce96","Type":"ContainerDied","Data":"1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa"}
Oct 31 16:39:52.529648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.529571    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" event={"ID":"6761d701-7651-4a9f-b397-0f29df36ce96","Type":"ContainerDied","Data":"a283ced7cc58e6c82d688f719a7c6cfaa38cfab7ddd231746224cd52b5f14f58"}
Oct 31 16:39:52.529648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.529595    4519 scope.go:122] "RemoveContainer" containerID="1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa"
Oct 31 16:39:52.552118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.552089    4519 scope.go:122] "RemoveContainer" containerID="1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa"
Oct 31 16:39:52.552517 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:52.552494    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa\": container with ID starting with 1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa not found: ID does not exist" containerID="1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa"
Oct 31 16:39:52.552596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.552530    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa"} err="failed to get container status \"1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa\": rpc error: code = NotFound desc = could not find container \"1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa\": container with ID starting with 1089ff20c8301ba94f8ec290ff14de660dad5df25b6646704e742a74e993f4aa not found: ID does not exist"
Oct 31 16:39:52.688808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.688717    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"]
Oct 31 16:39:52.694281 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:52.694248    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n"]
Oct 31 16:39:53.020778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:53.020734    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-w1-4fe3a4-kserve-7998794479-bpt7n" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:39:53.304937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:53.304845    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:53.306864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:53.306831    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:53.321896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:53.321865    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:39:53.518868 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:53.518835    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"]
Oct 31 16:39:53.521895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:53.521870    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-636a7f-nz4rw"]
Oct 31 16:39:54.005725 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:39:54.005693    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:39:54.446622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:54.446544    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0abafe09-6ecb-4212-a595-6f1279746c61" path="/var/lib/kubelet/pods/0abafe09-6ecb-4212-a595-6f1279746c61/volumes"
Oct 31 16:39:54.447029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:54.447014    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" path="/var/lib/kubelet/pods/6761d701-7651-4a9f-b397-0f29df36ce96/volumes"
Oct 31 16:39:57.180390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:57.180347    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" podUID="ce91a06d-620e-4c7f-9317-313db927a913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:39:58.939512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.939475    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"]
Oct 31 16:39:58.940125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.940109    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main"
Oct 31 16:39:58.940172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.940126    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main"
Oct 31 16:39:58.940172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.940150    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ba368270-cdbc-419e-85cb-74a6e8a0fab7" containerName="payload-processing"
Oct 31 16:39:58.940172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.940157    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba368270-cdbc-419e-85cb-74a6e8a0fab7" containerName="payload-processing"
Oct 31 16:39:58.940311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.940298    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba368270-cdbc-419e-85cb-74a6e8a0fab7" containerName="payload-processing"
Oct 31 16:39:58.940348 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.940322    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6761d701-7651-4a9f-b397-0f29df36ce96" containerName="main"
Oct 31 16:39:58.945298 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:58.945277    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.051486 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.051432    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.051779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.051752    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.051902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.051792    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.051957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.051946    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.052044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.052020    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.052108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.052046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.052108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.052074    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.052263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.052131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.052263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.052155    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvl4\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.052607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.052583    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"]
Oct 31 16:39:59.153400 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153367    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153426    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153449    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153502    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153672    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvl4\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153773    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153823    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153850    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.153907 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153877    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.154186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153923    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.154186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.153964    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.154263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.154184    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.154485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.154440    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.155956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.155933    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.156042 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.156002    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.208627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.208550    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.208776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.208668    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvl4\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qr6gj\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.259810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.259773    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:39:59.680234 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:39:59.680205    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e91534f_e6fa_4967_a948_9fd10cd089a8.slice/crio-e560e5aa5f77f0893dcdeac2df95bbf1cd19b4c86b4a138a0b06ae38099a0b35 WatchSource:0}: Error finding container e560e5aa5f77f0893dcdeac2df95bbf1cd19b4c86b4a138a0b06ae38099a0b35: Status 404 returned error can't find the container with id e560e5aa5f77f0893dcdeac2df95bbf1cd19b4c86b4a138a0b06ae38099a0b35
Oct 31 16:39:59.680523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.680501    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"]
Oct 31 16:39:59.682913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.682879    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:59.683009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.682948    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:59.683009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:39:59.682975    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:40:00.628662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:00.628622    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" event={"ID":"1e91534f-e6fa-4967-a948-9fd10cd089a8","Type":"ContainerStarted","Data":"2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05"}
Oct 31 16:40:00.629180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:00.628683    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" event={"ID":"1e91534f-e6fa-4967-a948-9fd10cd089a8","Type":"ContainerStarted","Data":"e560e5aa5f77f0893dcdeac2df95bbf1cd19b4c86b4a138a0b06ae38099a0b35"}
Oct 31 16:40:00.810337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:00.810255    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podStartSLOduration=2.810237764 podStartE2EDuration="2.810237764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:00.807909094 +0000 UTC m=+2244.507838405" watchObservedRunningTime="2026-10-31 16:40:00.810237764 +0000 UTC m=+2244.510167076"
Oct 31 16:40:01.259922 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.259889    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:40:01.261792 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.261765    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:01.262032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.262000    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:01.626732 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.626652    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"]
Oct 31 16:40:01.630963 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.630936    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.735044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.735003    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:40:01.735239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.735115    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:40:01.735239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.735131    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:40:01.735239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.735144    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:40:01.738617 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.738591    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"]
Oct 31 16:40:01.754030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.754003    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.754171 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.754150    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.754391 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.754373    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnmh\" (UniqueName: \"kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.855957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.855918    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnmh\" (UniqueName: \"kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.856139 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.856035    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.856139 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.856069    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.856752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.856728    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.858603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.858582    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.910413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.910334    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnmh\" (UniqueName: \"kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh\") pod \"maas-api-revoke-sub-6896399a-mkl2z\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:01.945418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:01.945385    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:02.260720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.260683    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:02.260933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.260734    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:02.346440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.346412    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"]
Oct 31 16:40:02.347103 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:40:02.347076    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cce11a5_b03b_4066_bda6_47a95ec657c4.slice/crio-6bb55b0f94cb234f63292ba85e97be76122e5ca182a27e5f2a4dd9a3784c5265 WatchSource:0}: Error finding container 6bb55b0f94cb234f63292ba85e97be76122e5ca182a27e5f2a4dd9a3784c5265: Status 404 returned error can't find the container with id 6bb55b0f94cb234f63292ba85e97be76122e5ca182a27e5f2a4dd9a3784c5265
Oct 31 16:40:02.645053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.645014    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z" event={"ID":"4cce11a5-b03b-4066-bda6-47a95ec657c4","Type":"ContainerStarted","Data":"6bb55b0f94cb234f63292ba85e97be76122e5ca182a27e5f2a4dd9a3784c5265"}
Oct 31 16:40:02.663192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.663163    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"]
Oct 31 16:40:02.696054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.696018    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.719783 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.719756    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"]
Oct 31 16:40:02.791527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.791442    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plnj\" (UniqueName: \"kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.791737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.791709    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.791905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.791885    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.897524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.896329    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.897524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.896482    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.897524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.896732    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plnj\" (UniqueName: \"kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.897984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.897957    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.899167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.899145    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:02.951604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:02.951577    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plnj\" (UniqueName: \"kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:03.010379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.010333    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:03.261390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.261355    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:03.261613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.261410    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:03.410053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.410015    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"]
Oct 31 16:40:03.410173 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:40:03.410068    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d388967_7fbc_4da2_83db_ac5ec32909e3.slice/crio-c57b071d41d84e8a37b8a1871c8234e6c11f785404995aa6289f8e1d6ae3d943 WatchSource:0}: Error finding container c57b071d41d84e8a37b8a1871c8234e6c11f785404995aa6289f8e1d6ae3d943: Status 404 returned error can't find the container with id c57b071d41d84e8a37b8a1871c8234e6c11f785404995aa6289f8e1d6ae3d943
Oct 31 16:40:03.654755 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.654717    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t" event={"ID":"9d388967-7fbc-4da2-83db-ac5ec32909e3","Type":"ContainerStarted","Data":"c57b071d41d84e8a37b8a1871c8234e6c11f785404995aa6289f8e1d6ae3d943"}
Oct 31 16:40:03.656611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.656579    4519 generic.go:356] "Generic (PLEG): container finished" podID="4cce11a5-b03b-4066-bda6-47a95ec657c4" containerID="5447cbde274ddc82fc7e6cc2bf77af7d41c4865476b92e0058b71f52b7a2c4e7" exitCode=0
Oct 31 16:40:03.656828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:03.656796    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z" event={"ID":"4cce11a5-b03b-4066-bda6-47a95ec657c4","Type":"ContainerDied","Data":"5447cbde274ddc82fc7e6cc2bf77af7d41c4865476b92e0058b71f52b7a2c4e7"}
Oct 31 16:40:04.260584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.260552    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:04.260776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.260606    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:04.691347 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.691312    4519 generic.go:356] "Generic (PLEG): container finished" podID="9d388967-7fbc-4da2-83db-ac5ec32909e3" containerID="ccb5abfdb85e9411b25b29df10a2cde28844a2171e0c4fd58db48ea27fd6f7e2" exitCode=0
Oct 31 16:40:04.691895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.691517    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t" event={"ID":"9d388967-7fbc-4da2-83db-ac5ec32909e3","Type":"ContainerDied","Data":"ccb5abfdb85e9411b25b29df10a2cde28844a2171e0c4fd58db48ea27fd6f7e2"}
Oct 31 16:40:04.864840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.864812    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:04.926533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.926502    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh\" (UniqueName: \"kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh\") pod \"4cce11a5-b03b-4066-bda6-47a95ec657c4\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") "
Oct 31 16:40:04.926724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.926570    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca\") pod \"4cce11a5-b03b-4066-bda6-47a95ec657c4\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") "
Oct 31 16:40:04.926724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.926715    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca\") pod \"4cce11a5-b03b-4066-bda6-47a95ec657c4\" (UID: \"4cce11a5-b03b-4066-bda6-47a95ec657c4\") "
Oct 31 16:40:04.926992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.926965    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca" pod "4cce11a5-b03b-4066-bda6-47a95ec657c4" (UID: "4cce11a5-b03b-4066-bda6-47a95ec657c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:04.927433 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.927287    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:04.928704 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.928681    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca" pod "4cce11a5-b03b-4066-bda6-47a95ec657c4" (UID: "4cce11a5-b03b-4066-bda6-47a95ec657c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:04.928840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:04.928821    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh" pod "4cce11a5-b03b-4066-bda6-47a95ec657c4" (UID: "4cce11a5-b03b-4066-bda6-47a95ec657c4"). InnerVolumeSpecName "kube-api-access-cpnmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:05.028950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.028913    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnmh\" (UniqueName: \"kubernetes.io/projected/4cce11a5-b03b-4066-bda6-47a95ec657c4-kube-api-access-cpnmh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:05.028950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.028944    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cce11a5-b03b-4066-bda6-47a95ec657c4-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:05.260465 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.260422    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:05.260645 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.260485    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:05.701984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.701908    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"
Oct 31 16:40:05.701984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.701920    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z" event={"ID":"4cce11a5-b03b-4066-bda6-47a95ec657c4","Type":"ContainerDied","Data":"6bb55b0f94cb234f63292ba85e97be76122e5ca182a27e5f2a4dd9a3784c5265"}
Oct 31 16:40:05.701984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.701965    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb55b0f94cb234f63292ba85e97be76122e5ca182a27e5f2a4dd9a3784c5265"
Oct 31 16:40:05.866686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.866654    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:05.938789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.938755    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca\") pod \"9d388967-7fbc-4da2-83db-ac5ec32909e3\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") "
Oct 31 16:40:05.939004 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.938844    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca\") pod \"9d388967-7fbc-4da2-83db-ac5ec32909e3\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") "
Oct 31 16:40:05.939004 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.938958    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj\" (UniqueName: \"kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj\") pod \"9d388967-7fbc-4da2-83db-ac5ec32909e3\" (UID: \"9d388967-7fbc-4da2-83db-ac5ec32909e3\") "
Oct 31 16:40:05.939202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.939172    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca" pod "9d388967-7fbc-4da2-83db-ac5ec32909e3" (UID: "9d388967-7fbc-4da2-83db-ac5ec32909e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:05.939512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.939491    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:05.941008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.940983    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca" pod "9d388967-7fbc-4da2-83db-ac5ec32909e3" (UID: "9d388967-7fbc-4da2-83db-ac5ec32909e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:05.941124 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:05.940990    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj" pod "9d388967-7fbc-4da2-83db-ac5ec32909e3" (UID: "9d388967-7fbc-4da2-83db-ac5ec32909e3"). InnerVolumeSpecName "kube-api-access-5plnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:06.040428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.040395    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d388967-7fbc-4da2-83db-ac5ec32909e3-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.040428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.040426    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plnj\" (UniqueName: \"kubernetes.io/projected/9d388967-7fbc-4da2-83db-ac5ec32909e3-kube-api-access-5plnj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.260304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.260260    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:06.260530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.260320    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:06.711601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.711573    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"
Oct 31 16:40:06.712014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.711604    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t" event={"ID":"9d388967-7fbc-4da2-83db-ac5ec32909e3","Type":"ContainerDied","Data":"c57b071d41d84e8a37b8a1871c8234e6c11f785404995aa6289f8e1d6ae3d943"}
Oct 31 16:40:06.712014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:06.711699    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57b071d41d84e8a37b8a1871c8234e6c11f785404995aa6289f8e1d6ae3d943"
Oct 31 16:40:07.180909 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:07.180820    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" podUID="ce91a06d-620e-4c7f-9317-313db927a913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:40:07.260174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:07.260137    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qr6gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:40:07.260357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:07.260202    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:40:08.264911 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.264875    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:40:08.265285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.265216    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:40:08.265990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.265974    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:40:08.579921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.579895    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:40:08.679812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.679778    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk\" (UniqueName: \"kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk\") pod \"ce91a06d-620e-4c7f-9317-313db927a913\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") "
Oct 31 16:40:08.680006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.679881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume\") pod \"ce91a06d-620e-4c7f-9317-313db927a913\" (UID: \"ce91a06d-620e-4c7f-9317-313db927a913\") "
Oct 31 16:40:08.680263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.680241    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume" pod "ce91a06d-620e-4c7f-9317-313db927a913" (UID: "ce91a06d-620e-4c7f-9317-313db927a913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:08.681918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.681896    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk" pod "ce91a06d-620e-4c7f-9317-313db927a913" (UID: "ce91a06d-620e-4c7f-9317-313db927a913"). InnerVolumeSpecName "kube-api-access-sx4lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:08.747521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.747445    4519 generic.go:356] "Generic (PLEG): container finished" podID="ce91a06d-620e-4c7f-9317-313db927a913" containerID="6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37" exitCode=0
Oct 31 16:40:08.747709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.747540    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" event={"ID":"ce91a06d-620e-4c7f-9317-313db927a913","Type":"ContainerDied","Data":"6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37"}
Oct 31 16:40:08.747709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.747616    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5" event={"ID":"ce91a06d-620e-4c7f-9317-313db927a913","Type":"ContainerDied","Data":"6549e9a4ba137a382a61bf1494c550f2c0647a02e87165403a66f06fd5847322"}
Oct 31 16:40:08.747709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.747648    4519 scope.go:122] "RemoveContainer" containerID="6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37"
Oct 31 16:40:08.748414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.748066    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"
Oct 31 16:40:08.770592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.770566    4519 scope.go:122] "RemoveContainer" containerID="6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37"
Oct 31 16:40:08.770934 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:40:08.770913    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37\": container with ID starting with 6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37 not found: ID does not exist" containerID="6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37"
Oct 31 16:40:08.771024 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.770945    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37"} err="failed to get container status \"6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37\": rpc error: code = NotFound desc = could not find container \"6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37\": container with ID starting with 6ae3169a81a64b6a8d912c2c035bf4ccaaf39a32e84a9d08fd1abc8e69ad6d37 not found: ID does not exist"
Oct 31 16:40:08.782064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.782021    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91a06d-620e-4c7f-9317-313db927a913-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:08.782064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.782057    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4lk\" (UniqueName: \"kubernetes.io/projected/ce91a06d-620e-4c7f-9317-313db927a913-kube-api-access-sx4lk\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:08.905240 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.905149    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"]
Oct 31 16:40:08.909290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:08.909257    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4fe3a4-54d4d5b77f-vshw5"]
Oct 31 16:40:10.439499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:10.439445    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce91a06d-620e-4c7f-9317-313db927a913" path="/var/lib/kubelet/pods/ce91a06d-620e-4c7f-9317-313db927a913/volumes"
Oct 31 16:40:14.230142 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:14.230107    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"]
Oct 31 16:40:14.330742 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:14.330707    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"]
Oct 31 16:40:19.231223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.231178    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2" podUID="0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" containerName="payload-processing" containerID="cri-o://c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2" gracePeriod=25
Oct 31 16:40:19.331058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.331016    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2" podUID="20b59a33-3dac-49d9-9d72-b932840a530c" containerName="payload-pre-processing" containerID="cri-o://025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074" gracePeriod=25
Oct 31 16:40:19.497198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.497125    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:40:19.619127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.618943    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:40:19.635915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.635888    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd\" (UniqueName: \"kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd\") pod \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") "
Oct 31 16:40:19.636092 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.636084    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume\") pod \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\" (UID: \"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23\") "
Oct 31 16:40:19.636448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.636423    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume" pod "0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" (UID: "0d3a725f-5a5f-4bb7-9843-f0374fcb1b23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:19.637964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.637943    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd" pod "0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" (UID: "0d3a725f-5a5f-4bb7-9843-f0374fcb1b23"). InnerVolumeSpecName "kube-api-access-l8kfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:19.737870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.737839    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h\" (UniqueName: \"kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h\") pod \"20b59a33-3dac-49d9-9d72-b932840a530c\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") "
Oct 31 16:40:19.738081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.737882    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume\") pod \"20b59a33-3dac-49d9-9d72-b932840a530c\" (UID: \"20b59a33-3dac-49d9-9d72-b932840a530c\") "
Oct 31 16:40:19.738289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.738268    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume" pod "20b59a33-3dac-49d9-9d72-b932840a530c" (UID: "20b59a33-3dac-49d9-9d72-b932840a530c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:19.738551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.738534    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8kfd\" (UniqueName: \"kubernetes.io/projected/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-kube-api-access-l8kfd\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:19.738551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.738552    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20b59a33-3dac-49d9-9d72-b932840a530c-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:19.738705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.738561    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:19.740086 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.740063    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h" pod "20b59a33-3dac-49d9-9d72-b932840a530c" (UID: "20b59a33-3dac-49d9-9d72-b932840a530c"). InnerVolumeSpecName "kube-api-access-jj56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:19.834079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.833966    4519 generic.go:356] "Generic (PLEG): container finished" podID="0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" containerID="c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2" exitCode=0
Oct 31 16:40:19.834272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.834101    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"
Oct 31 16:40:19.834272 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.834156    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2" event={"ID":"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23","Type":"ContainerDied","Data":"c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2"}
Oct 31 16:40:19.834380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.834286    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2" event={"ID":"0d3a725f-5a5f-4bb7-9843-f0374fcb1b23","Type":"ContainerDied","Data":"43d45be75d3ed431d4b79dd8d7e06bc905abd9ea59fd6f3407cf72300ceee383"}
Oct 31 16:40:19.834380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.834317    4519 scope.go:122] "RemoveContainer" containerID="c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2"
Oct 31 16:40:19.836035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.836009    4519 generic.go:356] "Generic (PLEG): container finished" podID="20b59a33-3dac-49d9-9d72-b932840a530c" containerID="025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074" exitCode=0
Oct 31 16:40:19.836172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.836051    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2" event={"ID":"20b59a33-3dac-49d9-9d72-b932840a530c","Type":"ContainerDied","Data":"025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074"}
Oct 31 16:40:19.836172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.836074    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"
Oct 31 16:40:19.836172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.836092    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2" event={"ID":"20b59a33-3dac-49d9-9d72-b932840a530c","Type":"ContainerDied","Data":"6e73522743d9af01a16374014f0e93295940af3f020d06f06c14f770772eba01"}
Oct 31 16:40:19.839672 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.839646    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj56h\" (UniqueName: \"kubernetes.io/projected/20b59a33-3dac-49d9-9d72-b932840a530c-kube-api-access-jj56h\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:19.845206 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.845136    4519 scope.go:122] "RemoveContainer" containerID="c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2"
Oct 31 16:40:19.845608 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:40:19.845478    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2\": container with ID starting with c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2 not found: ID does not exist" containerID="c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2"
Oct 31 16:40:19.845608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.845510    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2"} err="failed to get container status \"c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2\": rpc error: code = NotFound desc = could not find container \"c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2\": container with ID starting with c51bc4cb087b5f9a37f66402092e5370a88f9409d0b3d177da07281d2e37d6f2 not found: ID does not exist"
Oct 31 16:40:19.845608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.845527    4519 scope.go:122] "RemoveContainer" containerID="025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074"
Oct 31 16:40:19.855467 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.855432    4519 scope.go:122] "RemoveContainer" containerID="025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074"
Oct 31 16:40:19.855805 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:40:19.855785    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074\": container with ID starting with 025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074 not found: ID does not exist" containerID="025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074"
Oct 31 16:40:19.855860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.855813    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074"} err="failed to get container status \"025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074\": rpc error: code = NotFound desc = could not find container \"025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074\": container with ID starting with 025e3873ea16f8459a0ba5e0c8d1a00c9d82ebe111cc2ad4c85c791092efb074 not found: ID does not exist"
Oct 31 16:40:19.990676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.990645    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"]
Oct 31 16:40:19.993745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:19.993717    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d04acd-d55c97f55-pp6z2"]
Oct 31 16:40:20.093931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.093853    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"]
Oct 31 16:40:20.099925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.099898    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d04acd-8c7fdcf55-6gff2"]
Oct 31 16:40:20.439758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.439680    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" path="/var/lib/kubelet/pods/0d3a725f-5a5f-4bb7-9843-f0374fcb1b23/volumes"
Oct 31 16:40:20.440123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.440113    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b59a33-3dac-49d9-9d72-b932840a530c" path="/var/lib/kubelet/pods/20b59a33-3dac-49d9-9d72-b932840a530c/volumes"
Oct 31 16:40:20.823752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.823721    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"]
Oct 31 16:40:20.824309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824295    4519 cpu_manager.go:414] "containerMap: removing container" podUID="20b59a33-3dac-49d9-9d72-b932840a530c" containerName="payload-pre-processing"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824310    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b59a33-3dac-49d9-9d72-b932840a530c" containerName="payload-pre-processing"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824324    4519 cpu_manager.go:414] "containerMap: removing container" podUID="9d388967-7fbc-4da2-83db-ac5ec32909e3" containerName="revoke-keys"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824329    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d388967-7fbc-4da2-83db-ac5ec32909e3" containerName="revoke-keys"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824346    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ce91a06d-620e-4c7f-9317-313db927a913" containerName="payload-pre-processing"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824351    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce91a06d-620e-4c7f-9317-313db927a913" containerName="payload-pre-processing"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824358    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4cce11a5-b03b-4066-bda6-47a95ec657c4" containerName="revoke-keys"
Oct 31 16:40:20.824362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824363    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cce11a5-b03b-4066-bda6-47a95ec657c4" containerName="revoke-keys"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824380    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" containerName="payload-processing"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824385    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" containerName="payload-processing"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824480    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b59a33-3dac-49d9-9d72-b932840a530c" containerName="payload-pre-processing"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824489    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d388967-7fbc-4da2-83db-ac5ec32909e3" containerName="revoke-keys"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824497    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce91a06d-620e-4c7f-9317-313db927a913" containerName="payload-pre-processing"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824503    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cce11a5-b03b-4066-bda6-47a95ec657c4" containerName="revoke-keys"
Oct 31 16:40:20.824615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.824512    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3a725f-5a5f-4bb7-9843-f0374fcb1b23" containerName="payload-processing"
Oct 31 16:40:20.829653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.829636    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:20.880608 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.880577    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"]
Oct 31 16:40:20.923340 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.923308    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:40:20.923547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.923379    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:40:20.923547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.923388    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:40:20.926054 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.926032    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:40:20.953418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.953374    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:20.953629 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.953597    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:20.953677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:20.953658    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2klh\" (UniqueName: \"kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.054652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.054618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.054864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.054687    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2klh\" (UniqueName: \"kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.054864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.054799    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.055425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.055405    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.057051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.057029    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.108870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.108801    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2klh\" (UniqueName: \"kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh\") pod \"maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.143706 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.143670    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:21.510655 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:40:21.510623    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca142586_ff7d_466d_9cd8_2337c1365df1.slice/crio-78ee795255d6fc36eb38299602a4d77f2aeaca435ad9d538a4e09011a98afe1c WatchSource:0}: Error finding container 78ee795255d6fc36eb38299602a4d77f2aeaca435ad9d538a4e09011a98afe1c: Status 404 returned error can't find the container with id 78ee795255d6fc36eb38299602a4d77f2aeaca435ad9d538a4e09011a98afe1c
Oct 31 16:40:21.511053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.511031    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"]
Oct 31 16:40:21.853256 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.853209    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7" event={"ID":"ca142586-ff7d-466d-9cd8-2337c1365df1","Type":"ContainerStarted","Data":"92910a8a95f44dea9133860c73c73df56319d3e4691e8f0516e655dacf316bd6"}
Oct 31 16:40:21.853435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.853273    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7" event={"ID":"ca142586-ff7d-466d-9cd8-2337c1365df1","Type":"ContainerStarted","Data":"78ee795255d6fc36eb38299602a4d77f2aeaca435ad9d538a4e09011a98afe1c"}
Oct 31 16:40:21.961650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:21.961585    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7" podStartSLOduration=1.9615639759999999 podStartE2EDuration="1.961563976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:21.959592451 +0000 UTC m=+2265.659521764" watchObservedRunningTime="2026-10-31 16:40:21.961563976 +0000 UTC m=+2265.661493290"
Oct 31 16:40:22.862521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:22.862482    4519 generic.go:356] "Generic (PLEG): container finished" podID="ca142586-ff7d-466d-9cd8-2337c1365df1" containerID="92910a8a95f44dea9133860c73c73df56319d3e4691e8f0516e655dacf316bd6" exitCode=0
Oct 31 16:40:22.862521 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:22.862499    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7" event={"ID":"ca142586-ff7d-466d-9cd8-2337c1365df1","Type":"ContainerDied","Data":"92910a8a95f44dea9133860c73c73df56319d3e4691e8f0516e655dacf316bd6"}
Oct 31 16:40:24.007178 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:40:24.007145    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:40:24.051749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.051721    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:24.098437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.097028    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh\" (UniqueName: \"kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh\") pod \"ca142586-ff7d-466d-9cd8-2337c1365df1\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") "
Oct 31 16:40:24.098437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.097208    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca\") pod \"ca142586-ff7d-466d-9cd8-2337c1365df1\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") "
Oct 31 16:40:24.098437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.097308    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca\") pod \"ca142586-ff7d-466d-9cd8-2337c1365df1\" (UID: \"ca142586-ff7d-466d-9cd8-2337c1365df1\") "
Oct 31 16:40:24.100180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.100152    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca" pod "ca142586-ff7d-466d-9cd8-2337c1365df1" (UID: "ca142586-ff7d-466d-9cd8-2337c1365df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:24.116788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.116749    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh" pod "ca142586-ff7d-466d-9cd8-2337c1365df1" (UID: "ca142586-ff7d-466d-9cd8-2337c1365df1"). InnerVolumeSpecName "kube-api-access-j2klh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:24.121349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.121319    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca" pod "ca142586-ff7d-466d-9cd8-2337c1365df1" (UID: "ca142586-ff7d-466d-9cd8-2337c1365df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:24.199022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.198978    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:24.199022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.199021    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2klh\" (UniqueName: \"kubernetes.io/projected/ca142586-ff7d-466d-9cd8-2337c1365df1-kube-api-access-j2klh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:24.199303 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.199038    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca142586-ff7d-466d-9cd8-2337c1365df1-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:24.879840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.879812    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"
Oct 31 16:40:24.880001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.879844    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7" event={"ID":"ca142586-ff7d-466d-9cd8-2337c1365df1","Type":"ContainerDied","Data":"78ee795255d6fc36eb38299602a4d77f2aeaca435ad9d538a4e09011a98afe1c"}
Oct 31 16:40:24.880001 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:24.879891    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78ee795255d6fc36eb38299602a4d77f2aeaca435ad9d538a4e09011a98afe1c"
Oct 31 16:40:25.289364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:25.289322    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"]
Oct 31 16:40:28.247293 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.247254    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"]
Oct 31 16:40:28.248357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.248323    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ca142586-ff7d-466d-9cd8-2337c1365df1" containerName="revoke-keys"
Oct 31 16:40:28.248357 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.248348    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca142586-ff7d-466d-9cd8-2337c1365df1" containerName="revoke-keys"
Oct 31 16:40:28.248537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.248503    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca142586-ff7d-466d-9cd8-2337c1365df1" containerName="revoke-keys"
Oct 31 16:40:28.252355 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.252335    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.304280 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.304245    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"]
Oct 31 16:40:28.346520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.346469    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:40:28.346520 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.346450    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:40:28.346804 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.346591    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:40:28.350053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.350020    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"]
Oct 31 16:40:28.351885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.351863    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:40:28.354354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.354333    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.355984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.355963    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.356110 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.356093    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.356251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.356229    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv8kj\" (UniqueName: \"kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.365083 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.365056    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"]
Oct 31 16:40:28.457609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.457577    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.457818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.457644    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw4x\" (UniqueName: \"kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.457818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.457747    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv8kj\" (UniqueName: \"kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.457818 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.457809    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.458006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.457828    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.458006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.457956    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.458554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.458537    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.460025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.460005    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.515002 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.514925    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv8kj\" (UniqueName: \"kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj\") pod \"maas-api-revoke-sub-786e30b9-89scl\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.559692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.559656    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw4x\" (UniqueName: \"kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.559895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.559741    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.559895 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.559759    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.560434 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.560406    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.562095 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.562070    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.567098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.567080    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:28.616606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.616576    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw4x\" (UniqueName: \"kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x\") pod \"maas-api-revoke-sub-fce78cad-zgcg9\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.671849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.671819    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:28.944239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:28.944207    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"]
Oct 31 16:40:28.944355 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:40:28.944304    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6f27f6_5ed8_4493_97d2_72d86457e796.slice/crio-d795e9047e3b041ce6450359cc637b9367f8251908307f87b1dd8e511fcc5ba8 WatchSource:0}: Error finding container d795e9047e3b041ce6450359cc637b9367f8251908307f87b1dd8e511fcc5ba8: Status 404 returned error can't find the container with id d795e9047e3b041ce6450359cc637b9367f8251908307f87b1dd8e511fcc5ba8
Oct 31 16:40:29.049232 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:40:29.049198    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449de57b_fa3e_4296_b562_9002efea6e4e.slice/crio-6700d80f4b724c3002238f5716c0cc221b225cb1602dc724873c32dfff4adfb6 WatchSource:0}: Error finding container 6700d80f4b724c3002238f5716c0cc221b225cb1602dc724873c32dfff4adfb6: Status 404 returned error can't find the container with id 6700d80f4b724c3002238f5716c0cc221b225cb1602dc724873c32dfff4adfb6
Oct 31 16:40:29.049335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:29.049303    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"]
Oct 31 16:40:29.917809 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:29.917767    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl" event={"ID":"ca6f27f6-5ed8-4493-97d2-72d86457e796","Type":"ContainerStarted","Data":"94789620f5d23de87a6dd0921ced167e02cb2cdf863c1d53d0bb9651768e6c48"}
Oct 31 16:40:29.918270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:29.917836    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl" event={"ID":"ca6f27f6-5ed8-4493-97d2-72d86457e796","Type":"ContainerStarted","Data":"d795e9047e3b041ce6450359cc637b9367f8251908307f87b1dd8e511fcc5ba8"}
Oct 31 16:40:29.919511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:29.919483    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9" event={"ID":"449de57b-fa3e-4296-b562-9002efea6e4e","Type":"ContainerStarted","Data":"b892fd332a140817dcdbb8ff6290a951253bace450db182dc3a915d939fef478"}
Oct 31 16:40:29.919632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:29.919537    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9" event={"ID":"449de57b-fa3e-4296-b562-9002efea6e4e","Type":"ContainerStarted","Data":"6700d80f4b724c3002238f5716c0cc221b225cb1602dc724873c32dfff4adfb6"}
Oct 31 16:40:30.025294 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:30.025242    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl" podStartSLOduration=2.025224966 podStartE2EDuration="2.025224966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:30.024740224 +0000 UTC m=+2273.724669543" watchObservedRunningTime="2026-10-31 16:40:30.025224966 +0000 UTC m=+2273.725154274"
Oct 31 16:40:30.134468 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:30.134402    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9" podStartSLOduration=2.134385782 podStartE2EDuration="2.134385782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:30.131838159 +0000 UTC m=+2273.831767471" watchObservedRunningTime="2026-10-31 16:40:30.134385782 +0000 UTC m=+2273.834315135"
Oct 31 16:40:30.928657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:30.928619    4519 generic.go:356] "Generic (PLEG): container finished" podID="ca6f27f6-5ed8-4493-97d2-72d86457e796" containerID="94789620f5d23de87a6dd0921ced167e02cb2cdf863c1d53d0bb9651768e6c48" exitCode=0
Oct 31 16:40:30.929091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:30.928705    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl" event={"ID":"ca6f27f6-5ed8-4493-97d2-72d86457e796","Type":"ContainerDied","Data":"94789620f5d23de87a6dd0921ced167e02cb2cdf863c1d53d0bb9651768e6c48"}
Oct 31 16:40:30.930398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:30.930375    4519 generic.go:356] "Generic (PLEG): container finished" podID="449de57b-fa3e-4296-b562-9002efea6e4e" containerID="b892fd332a140817dcdbb8ff6290a951253bace450db182dc3a915d939fef478" exitCode=0
Oct 31 16:40:30.930527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:30.930404    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9" event={"ID":"449de57b-fa3e-4296-b562-9002efea6e4e","Type":"ContainerDied","Data":"b892fd332a140817dcdbb8ff6290a951253bace450db182dc3a915d939fef478"}
Oct 31 16:40:32.116870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.116843    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:32.137035 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.137014    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:32.208861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.208781    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj\" (UniqueName: \"kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj\") pod \"ca6f27f6-5ed8-4493-97d2-72d86457e796\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") "
Oct 31 16:40:32.208861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.208836    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca\") pod \"ca6f27f6-5ed8-4493-97d2-72d86457e796\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") "
Oct 31 16:40:32.209080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.208978    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca\") pod \"449de57b-fa3e-4296-b562-9002efea6e4e\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") "
Oct 31 16:40:32.209136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209102    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca\") pod \"ca6f27f6-5ed8-4493-97d2-72d86457e796\" (UID: \"ca6f27f6-5ed8-4493-97d2-72d86457e796\") "
Oct 31 16:40:32.209189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209140    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x\" (UniqueName: \"kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x\") pod \"449de57b-fa3e-4296-b562-9002efea6e4e\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") "
Oct 31 16:40:32.209189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209178    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca\") pod \"449de57b-fa3e-4296-b562-9002efea6e4e\" (UID: \"449de57b-fa3e-4296-b562-9002efea6e4e\") "
Oct 31 16:40:32.209394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209273    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca" pod "ca6f27f6-5ed8-4493-97d2-72d86457e796" (UID: "ca6f27f6-5ed8-4493-97d2-72d86457e796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:32.209674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209638    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca" pod "449de57b-fa3e-4296-b562-9002efea6e4e" (UID: "449de57b-fa3e-4296-b562-9002efea6e4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:32.209849 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209828    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:32.209906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.209854    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:32.211278 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.211256    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca" pod "ca6f27f6-5ed8-4493-97d2-72d86457e796" (UID: "ca6f27f6-5ed8-4493-97d2-72d86457e796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:32.211367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.211293    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x" pod "449de57b-fa3e-4296-b562-9002efea6e4e" (UID: "449de57b-fa3e-4296-b562-9002efea6e4e"). InnerVolumeSpecName "kube-api-access-vmw4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:32.211590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.211563    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca" pod "449de57b-fa3e-4296-b562-9002efea6e4e" (UID: "449de57b-fa3e-4296-b562-9002efea6e4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:32.211684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.211566    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj" pod "ca6f27f6-5ed8-4493-97d2-72d86457e796" (UID: "ca6f27f6-5ed8-4493-97d2-72d86457e796"). InnerVolumeSpecName "kube-api-access-tv8kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:32.311570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.311534    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca6f27f6-5ed8-4493-97d2-72d86457e796-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:32.311570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.311568    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmw4x\" (UniqueName: \"kubernetes.io/projected/449de57b-fa3e-4296-b562-9002efea6e4e-kube-api-access-vmw4x\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:32.311570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.311580    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv8kj\" (UniqueName: \"kubernetes.io/projected/ca6f27f6-5ed8-4493-97d2-72d86457e796-kube-api-access-tv8kj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:32.311570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.311589    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449de57b-fa3e-4296-b562-9002efea6e4e-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:32.946951 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.946919    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"
Oct 31 16:40:32.947148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.946913    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl" event={"ID":"ca6f27f6-5ed8-4493-97d2-72d86457e796","Type":"ContainerDied","Data":"d795e9047e3b041ce6450359cc637b9367f8251908307f87b1dd8e511fcc5ba8"}
Oct 31 16:40:32.947148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.947065    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d795e9047e3b041ce6450359cc637b9367f8251908307f87b1dd8e511fcc5ba8"
Oct 31 16:40:32.948513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.948491    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"
Oct 31 16:40:32.948513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.948503    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9" event={"ID":"449de57b-fa3e-4296-b562-9002efea6e4e","Type":"ContainerDied","Data":"6700d80f4b724c3002238f5716c0cc221b225cb1602dc724873c32dfff4adfb6"}
Oct 31 16:40:32.948687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:32.948543    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6700d80f4b724c3002238f5716c0cc221b225cb1602dc724873c32dfff4adfb6"
Oct 31 16:40:35.828627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:35.828593    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"]
Oct 31 16:40:35.875878 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:35.875847    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7f13a-lwzv8"]
Oct 31 16:40:36.441502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:36.441442    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125915c2-a89e-4ba0-9bdf-cb93c3f75b7b" path="/var/lib/kubelet/pods/125915c2-a89e-4ba0-9bdf-cb93c3f75b7b/volumes"
Oct 31 16:40:38.696925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:38.696892    4519 scope.go:122] "RemoveContainer" containerID="6511a0bc64755220f2aeaab5e3bfe25b173e267173b4fd5b0f9913b550d1cf82"
Oct 31 16:40:38.708840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:38.708820    4519 scope.go:122] "RemoveContainer" containerID="1bd0a8b83d88f2eb1afe89c9d791ba9f2874e90d90eeaa76a05e5ce4d9d70f11"
Oct 31 16:40:40.037777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.037744    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"]
Oct 31 16:40:40.038373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.038357    4519 cpu_manager.go:414] "containerMap: removing container" podUID="449de57b-fa3e-4296-b562-9002efea6e4e" containerName="revoke-keys"
Oct 31 16:40:40.038423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.038374    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449de57b-fa3e-4296-b562-9002efea6e4e" containerName="revoke-keys"
Oct 31 16:40:40.038423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.038398    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ca6f27f6-5ed8-4493-97d2-72d86457e796" containerName="revoke-keys"
Oct 31 16:40:40.038423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.038403    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6f27f6-5ed8-4493-97d2-72d86457e796" containerName="revoke-keys"
Oct 31 16:40:40.038603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.038512    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="449de57b-fa3e-4296-b562-9002efea6e4e" containerName="revoke-keys"
Oct 31 16:40:40.038603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.038523    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6f27f6-5ed8-4493-97d2-72d86457e796" containerName="revoke-keys"
Oct 31 16:40:40.043872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.043852    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.089481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.089437    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"]
Oct 31 16:40:40.094501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.094475    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.096696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.096668    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"]
Oct 31 16:40:40.147537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.147495    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"]
Oct 31 16:40:40.147747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.147670    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:40:40.147829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.147754    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:40:40.147829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.147799    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:40:40.147950 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.147906    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:40:40.169579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.169537    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.169778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.169671    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.169778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.169729    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphnb\" (UniqueName: \"kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.169927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.169784    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5f45\" (UniqueName: \"kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.169927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.169804    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.169927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.169907    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.272369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.272330    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.272613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.272429    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.272613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.272504    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphnb\" (UniqueName: \"kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.272613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.272540    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5f45\" (UniqueName: \"kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.272613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.272566    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.272861 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.272657    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.273093 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.273069    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.273163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.273123    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.275112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.275084    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.275218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.275084    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.310959 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.310839    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main" containerID="cri-o://a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5" gracePeriod=45
Oct 31 16:40:40.327113 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.327079    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphnb\" (UniqueName: \"kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb\") pod \"maas-api-revoke-sub-2f5f8567-qhrzz\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.329136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.329113    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5f45\" (UniqueName: \"kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45\") pod \"maas-api-revoke-sub-3b84ba9f-m8gzp\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.357261 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.357221    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:40.413949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.410816    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:40.725615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.725585    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:40:40.734551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.734524    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"]
Oct 31 16:40:40.779022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.778989    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home\") pod \"667e8d38-e5ab-4af8-a35a-def8a1083241\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") "
Oct 31 16:40:40.779187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779032    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache\") pod \"667e8d38-e5ab-4af8-a35a-def8a1083241\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") "
Oct 31 16:40:40.779187 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779072    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm\") pod \"667e8d38-e5ab-4af8-a35a-def8a1083241\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") "
Oct 31 16:40:40.779311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779221    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home" pod "667e8d38-e5ab-4af8-a35a-def8a1083241" (UID: "667e8d38-e5ab-4af8-a35a-def8a1083241"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.779311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779268    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir\") pod \"667e8d38-e5ab-4af8-a35a-def8a1083241\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") "
Oct 31 16:40:40.779402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779323    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb\" (UniqueName: \"kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb\") pod \"667e8d38-e5ab-4af8-a35a-def8a1083241\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") "
Oct 31 16:40:40.779402 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779360    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache" pod "667e8d38-e5ab-4af8-a35a-def8a1083241" (UID: "667e8d38-e5ab-4af8-a35a-def8a1083241"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.779491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779408    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs\" (UniqueName: \"kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs\") pod \"667e8d38-e5ab-4af8-a35a-def8a1083241\" (UID: \"667e8d38-e5ab-4af8-a35a-def8a1083241\") "
Oct 31 16:40:40.779536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779518    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir" pod "667e8d38-e5ab-4af8-a35a-def8a1083241" (UID: "667e8d38-e5ab-4af8-a35a-def8a1083241"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.779890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779860    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.779890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779885    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.780084 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.779897    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.781428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.781408    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs" pod "667e8d38-e5ab-4af8-a35a-def8a1083241" (UID: "667e8d38-e5ab-4af8-a35a-def8a1083241"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:40.781428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.781419    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm" pod "667e8d38-e5ab-4af8-a35a-def8a1083241" (UID: "667e8d38-e5ab-4af8-a35a-def8a1083241"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.781543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.781437    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb" pod "667e8d38-e5ab-4af8-a35a-def8a1083241" (UID: "667e8d38-e5ab-4af8-a35a-def8a1083241"). InnerVolumeSpecName "kube-api-access-ct8kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:40.839343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.839293    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"]
Oct 31 16:40:40.840820 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:40:40.840781    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf465efc6_db89_4a9e_b02e_5cc199f72782.slice/crio-53591a7f50243d8ceb73ce6e97c3f15f948e2fefec27b2a7060abea465329a77 WatchSource:0}: Error finding container 53591a7f50243d8ceb73ce6e97c3f15f948e2fefec27b2a7060abea465329a77: Status 404 returned error can't find the container with id 53591a7f50243d8ceb73ce6e97c3f15f948e2fefec27b2a7060abea465329a77
Oct 31 16:40:40.885739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.881436    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct8kb\" (UniqueName: \"kubernetes.io/projected/667e8d38-e5ab-4af8-a35a-def8a1083241-kube-api-access-ct8kb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.885739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.881485    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e8d38-e5ab-4af8-a35a-def8a1083241-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.885739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.881499    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e8d38-e5ab-4af8-a35a-def8a1083241-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.962765 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:40.962734    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:40:41.022560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.022514    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz" event={"ID":"f465efc6-db89-4a9e-b02e-5cc199f72782","Type":"ContainerStarted","Data":"53591a7f50243d8ceb73ce6e97c3f15f948e2fefec27b2a7060abea465329a77"}
Oct 31 16:40:41.024290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.024242    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp" event={"ID":"208030bb-9d6a-4b0a-91f6-e81fd8e615b5","Type":"ContainerStarted","Data":"d386411c275b2aab7a05972d525788e839b7ad3314487f7f8d455aa92373d158"}
Oct 31 16:40:41.025851 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.025826    4519 generic.go:356] "Generic (PLEG): container finished" podID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerID="a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5" exitCode=0
Oct 31 16:40:41.025983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.025875    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"
Oct 31 16:40:41.025983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.025907    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" event={"ID":"667e8d38-e5ab-4af8-a35a-def8a1083241","Type":"ContainerDied","Data":"a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5"}
Oct 31 16:40:41.025983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.025953    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4" event={"ID":"667e8d38-e5ab-4af8-a35a-def8a1083241","Type":"ContainerDied","Data":"22314caefce78ab1167e83b8e1851d48851d68792d9b931eb3510abf795f48cc"}
Oct 31 16:40:41.025983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.025974    4519 scope.go:122] "RemoveContainer" containerID="a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5"
Oct 31 16:40:41.036767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.036463    4519 scope.go:122] "RemoveContainer" containerID="a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5"
Oct 31 16:40:41.037155 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:40:41.037130    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5\": container with ID starting with a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5 not found: ID does not exist" containerID="a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5"
Oct 31 16:40:41.037233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.037166    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5"} err="failed to get container status \"a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5\": rpc error: code = NotFound desc = could not find container \"a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5\": container with ID starting with a81b113524712b120f2c7fb7caba9e2ea6ddf6bcdb45821daa59fcec701f97d5 not found: ID does not exist"
Oct 31 16:40:41.209536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.209503    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"]
Oct 31 16:40:41.218918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:41.218841    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-2-w4-59357a-kserve-769dff4844-m7fl4"]
Oct 31 16:40:42.064819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:42.062994    4519 generic.go:356] "Generic (PLEG): container finished" podID="208030bb-9d6a-4b0a-91f6-e81fd8e615b5" containerID="71180388ef09bf5b0d798b33992b31082dc5df264d264b3946905fabfa2b06af" exitCode=0
Oct 31 16:40:42.064819 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:42.063111    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp" event={"ID":"208030bb-9d6a-4b0a-91f6-e81fd8e615b5","Type":"ContainerDied","Data":"71180388ef09bf5b0d798b33992b31082dc5df264d264b3946905fabfa2b06af"}
Oct 31 16:40:42.068053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:42.068019    4519 generic.go:356] "Generic (PLEG): container finished" podID="f465efc6-db89-4a9e-b02e-5cc199f72782" containerID="42d8e2ce3ab93080af40fcb69b2536a7adf78222b2cb506553fc804c2906cfef" exitCode=0
Oct 31 16:40:42.068330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:42.068309    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz" event={"ID":"f465efc6-db89-4a9e-b02e-5cc199f72782","Type":"ContainerDied","Data":"42d8e2ce3ab93080af40fcb69b2536a7adf78222b2cb506553fc804c2906cfef"}
Oct 31 16:40:42.444308 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:42.444261    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" path="/var/lib/kubelet/pods/667e8d38-e5ab-4af8-a35a-def8a1083241/volumes"
Oct 31 16:40:43.751483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:43.692974    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"]
Oct 31 16:40:43.751483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:43.693259    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy" containerID="cri-o://9d6bddee6e568fd63b6dfd8f35cbb20e3984f84c15592b4a06c3a4deab3e3c19" gracePeriod=30
Oct 31 16:40:43.958186 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:43.958150    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:44.015060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.012879    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb\" (UniqueName: \"kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb\") pod \"f465efc6-db89-4a9e-b02e-5cc199f72782\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") "
Oct 31 16:40:44.015640 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.015612    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca\") pod \"f465efc6-db89-4a9e-b02e-5cc199f72782\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") "
Oct 31 16:40:44.015913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.015895    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca\") pod \"f465efc6-db89-4a9e-b02e-5cc199f72782\" (UID: \"f465efc6-db89-4a9e-b02e-5cc199f72782\") "
Oct 31 16:40:44.020923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.020398    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca" pod "f465efc6-db89-4a9e-b02e-5cc199f72782" (UID: "f465efc6-db89-4a9e-b02e-5cc199f72782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:44.022720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.021838    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca" pod "f465efc6-db89-4a9e-b02e-5cc199f72782" (UID: "f465efc6-db89-4a9e-b02e-5cc199f72782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:44.028176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.023641    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb" pod "f465efc6-db89-4a9e-b02e-5cc199f72782" (UID: "f465efc6-db89-4a9e-b02e-5cc199f72782"). InnerVolumeSpecName "kube-api-access-bphnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:44.058601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.058567    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:44.141518 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.141477    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca\") pod \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") "
Oct 31 16:40:44.141708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.141575    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca\") pod \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") "
Oct 31 16:40:44.141802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.141786    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45\" (UniqueName: \"kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45\") pod \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\" (UID: \"208030bb-9d6a-4b0a-91f6-e81fd8e615b5\") "
Oct 31 16:40:44.141985 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.141960    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca" pod "208030bb-9d6a-4b0a-91f6-e81fd8e615b5" (UID: "208030bb-9d6a-4b0a-91f6-e81fd8e615b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:44.142395 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.142341    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:44.142631 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.142611    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bphnb\" (UniqueName: \"kubernetes.io/projected/f465efc6-db89-4a9e-b02e-5cc199f72782-kube-api-access-bphnb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:44.142731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.142719    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:44.142812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.142802    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f465efc6-db89-4a9e-b02e-5cc199f72782-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:44.144751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.144720    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45" pod "208030bb-9d6a-4b0a-91f6-e81fd8e615b5" (UID: "208030bb-9d6a-4b0a-91f6-e81fd8e615b5"). InnerVolumeSpecName "kube-api-access-t5f45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:44.145691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.145569    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca" pod "208030bb-9d6a-4b0a-91f6-e81fd8e615b5" (UID: "208030bb-9d6a-4b0a-91f6-e81fd8e615b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:44.147958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.147798    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp" event={"ID":"208030bb-9d6a-4b0a-91f6-e81fd8e615b5","Type":"ContainerDied","Data":"d386411c275b2aab7a05972d525788e839b7ad3314487f7f8d455aa92373d158"}
Oct 31 16:40:44.147958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.147858    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d386411c275b2aab7a05972d525788e839b7ad3314487f7f8d455aa92373d158"
Oct 31 16:40:44.147958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.147934    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"
Oct 31 16:40:44.150756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.150727    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz" event={"ID":"f465efc6-db89-4a9e-b02e-5cc199f72782","Type":"ContainerDied","Data":"53591a7f50243d8ceb73ce6e97c3f15f948e2fefec27b2a7060abea465329a77"}
Oct 31 16:40:44.150879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.150760    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"
Oct 31 16:40:44.150879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.150774    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53591a7f50243d8ceb73ce6e97c3f15f948e2fefec27b2a7060abea465329a77"
Oct 31 16:40:44.245052 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.245014    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5f45\" (UniqueName: \"kubernetes.io/projected/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-kube-api-access-t5f45\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:44.245052 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.245053    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208030bb-9d6a-4b0a-91f6-e81fd8e615b5-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:44.906551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.905213    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"]
Oct 31 16:40:44.906551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.905490    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy" containerID="cri-o://e245c6d45b707b85b8e3e1b28e812f82a26eaa808d559a0fb20d759164a0f7ac" gracePeriod=30
Oct 31 16:40:44.908731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.908705    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"]
Oct 31 16:40:44.909044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.909009    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy" containerID="cri-o://886177cbfe060752d887d7ecf70f37ce6d1b0456cd86e2996889bbf155e12d8a" gracePeriod=30
Oct 31 16:40:44.923530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.923500    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"]
Oct 31 16:40:44.923770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:44.923738    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy" containerID="cri-o://04b85901aaf1c73a1843c955deb70d6102822f7594f7d53e810060bdf23d2fa8" gracePeriod=30
Oct 31 16:40:49.195318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.195232    4519 generic.go:356] "Generic (PLEG): container finished" podID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerID="9d6bddee6e568fd63b6dfd8f35cbb20e3984f84c15592b4a06c3a4deab3e3c19" exitCode=0
Oct 31 16:40:49.195318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.195301    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" event={"ID":"6f39b36f-6f3f-4081-9e40-de9dbff3c623","Type":"ContainerDied","Data":"9d6bddee6e568fd63b6dfd8f35cbb20e3984f84c15592b4a06c3a4deab3e3c19"}
Oct 31 16:40:49.195743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.195360    4519 scope.go:122] "RemoveContainer" containerID="0aeb0c1d0d2bb18fdf4990455db1fe789c3a97ff2e998b86cbe66b0dd7c75d19"
Oct 31 16:40:49.581485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.581434    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:40:49.621667 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.621632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.621860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.621797    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.621928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.621884    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.621986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.621931    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.622044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.621996    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.622120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622043    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.622120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622086    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.622120 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622086    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:49.622290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622164    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.622290 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622206    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket\") pod \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\" (UID: \"6f39b36f-6f3f-4081-9e40-de9dbff3c623\") "
Oct 31 16:40:49.622409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622310    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:49.622409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622332    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:49.622687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622654    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:49.622687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622667    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:49.622933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622912    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.623018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622937    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.623018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622952    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.623018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622964    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.623018 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.622978    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.624279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.624254    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:49.624412 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.624363    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:49.624565 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.624534    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "kube-api-access-vwq4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:49.624668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.624611    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token" pod "6f39b36f-6f3f-4081-9e40-de9dbff3c623" (UID: "6f39b36f-6f3f-4081-9e40-de9dbff3c623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:49.723997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.723961    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwq4f\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-kube-api-access-vwq4f\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.723997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.723990    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.723997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.724005    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.724279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:49.724017    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f39b36f-6f3f-4081-9e40-de9dbff3c623-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.207805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.207771    4519 generic.go:356] "Generic (PLEG): container finished" podID="e0d6b028-b826-4c73-8625-2c3291adc268" containerID="e245c6d45b707b85b8e3e1b28e812f82a26eaa808d559a0fb20d759164a0f7ac" exitCode=0
Oct 31 16:40:50.208262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.207843    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" event={"ID":"e0d6b028-b826-4c73-8625-2c3291adc268","Type":"ContainerDied","Data":"e245c6d45b707b85b8e3e1b28e812f82a26eaa808d559a0fb20d759164a0f7ac"}
Oct 31 16:40:50.209758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.209712    4519 generic.go:356] "Generic (PLEG): container finished" podID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerID="886177cbfe060752d887d7ecf70f37ce6d1b0456cd86e2996889bbf155e12d8a" exitCode=0
Oct 31 16:40:50.209913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.209843    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" event={"ID":"8350bd2c-9c50-4fb6-bd89-28bab01a2aed","Type":"ContainerDied","Data":"886177cbfe060752d887d7ecf70f37ce6d1b0456cd86e2996889bbf155e12d8a"}
Oct 31 16:40:50.209913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.209890    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h" event={"ID":"8350bd2c-9c50-4fb6-bd89-28bab01a2aed","Type":"ContainerDied","Data":"e6af621761a56b4cbc5123996faf6cbaa8917b1911941a4262aba61258c557b1"}
Oct 31 16:40:50.210021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.209914    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6af621761a56b4cbc5123996faf6cbaa8917b1911941a4262aba61258c557b1"
Oct 31 16:40:50.211778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.211751    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g" event={"ID":"6f39b36f-6f3f-4081-9e40-de9dbff3c623","Type":"ContainerDied","Data":"3c245a093e9f464298cb32acbef319f8d3a03147fcbb4c344f10e24442e46058"}
Oct 31 16:40:50.211778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.211776    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"
Oct 31 16:40:50.211952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.211820    4519 scope.go:122] "RemoveContainer" containerID="9d6bddee6e568fd63b6dfd8f35cbb20e3984f84c15592b4a06c3a4deab3e3c19"
Oct 31 16:40:50.212371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.212337    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:40:50.214904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.214712    4519 generic.go:356] "Generic (PLEG): container finished" podID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerID="04b85901aaf1c73a1843c955deb70d6102822f7594f7d53e810060bdf23d2fa8" exitCode=0
Oct 31 16:40:50.214904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.214835    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" event={"ID":"7a63eb5e-a2ec-4b10-917e-23c984e5d292","Type":"ContainerDied","Data":"04b85901aaf1c73a1843c955deb70d6102822f7594f7d53e810060bdf23d2fa8"}
Oct 31 16:40:50.238545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.238522    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:40:50.249987 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.249965    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:40:50.336676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336637    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.336676 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.336939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336703    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.336939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336747    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.336939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336780    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.336939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336825    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.336939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336861    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.336939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336891    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336962    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.336981    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337090    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337097    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337162    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337170    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337213    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.337270 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337242    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337272    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337289    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337310    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337339    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337341    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337480    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337575    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337483    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337613    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337660    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337701    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337733    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.337757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337761    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337797    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337802    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337835    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337868    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo\") pod \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\" (UID: \"8350bd2c-9c50-4fb6-bd89-28bab01a2aed\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337906    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337946    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.337985    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data\") pod \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\" (UID: \"7a63eb5e-a2ec-4b10-917e-23c984e5d292\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338027    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert\") pod \"e0d6b028-b826-4c73-8625-2c3291adc268\" (UID: \"e0d6b028-b826-4c73-8625-2c3291adc268\") "
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338105    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338421    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338666    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338751    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338804    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338823    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.338834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338838    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338852    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338866    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338905    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338920    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338934    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338945    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338958    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338971    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338986    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.338994    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.339739 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.339002    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.340387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.339991    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.340387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.340094    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "kube-api-access-vvlbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.340387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.340148    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.340387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.340320    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.340775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.340747    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "kube-api-access-fkqvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.341037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.341012    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.341417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.341389    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.341677 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.341660    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.342316 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.342292    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f" pod "8350bd2c-9c50-4fb6-bd89-28bab01a2aed" (UID: "8350bd2c-9c50-4fb6-bd89-28bab01a2aed"). InnerVolumeSpecName "kube-api-access-fgh5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.342615 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.342593    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.342767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.342750    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token" pod "7a63eb5e-a2ec-4b10-917e-23c984e5d292" (UID: "7a63eb5e-a2ec-4b10-917e-23c984e5d292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.342846 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.342828    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token" pod "e0d6b028-b826-4c73-8625-2c3291adc268" (UID: "e0d6b028-b826-4c73-8625-2c3291adc268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.440234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440156    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440192    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440208    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440225    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgh5f\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-kube-api-access-fgh5f\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440240    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440255    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d6b028-b826-4c73-8625-2c3291adc268-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440269    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440283    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a63eb5e-a2ec-4b10-917e-23c984e5d292-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440295    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d6b028-b826-4c73-8625-2c3291adc268-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440310    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d6b028-b826-4c73-8625-2c3291adc268-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440324    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440337    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8350bd2c-9c50-4fb6-bd89-28bab01a2aed-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440352    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvlbp\" (UniqueName: \"kubernetes.io/projected/e0d6b028-b826-4c73-8625-2c3291adc268-kube-api-access-vvlbp\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.440531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.440368    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqvp\" (UniqueName: \"kubernetes.io/projected/7a63eb5e-a2ec-4b10-917e-23c984e5d292-kube-api-access-fkqvp\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.697752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.697659    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"]
Oct 31 16:40:50.701087 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:50.701060    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vvx8g"]
Oct 31 16:40:51.222920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.222889    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"
Oct 31 16:40:51.222920 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.222904    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24" event={"ID":"7a63eb5e-a2ec-4b10-917e-23c984e5d292","Type":"ContainerDied","Data":"231cdf984e2a67a028f7f64c93cf5e63e3297b1a05e9b8c8f4ba09414b833035"}
Oct 31 16:40:51.223540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.223014    4519 scope.go:122] "RemoveContainer" containerID="04b85901aaf1c73a1843c955deb70d6102822f7594f7d53e810060bdf23d2fa8"
Oct 31 16:40:51.224820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.224799    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"
Oct 31 16:40:51.224943 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.224822    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw" event={"ID":"e0d6b028-b826-4c73-8625-2c3291adc268","Type":"ContainerDied","Data":"e9c34174aaf2db9ecf4dba9a358c131086a7b4666c885b4ccb9c29367f5413b8"}
Oct 31 16:40:51.226174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.226158    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"
Oct 31 16:40:51.235125 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.235107    4519 scope.go:122] "RemoveContainer" containerID="e245c6d45b707b85b8e3e1b28e812f82a26eaa808d559a0fb20d759164a0f7ac"
Oct 31 16:40:51.379763 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.379731    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"]
Oct 31 16:40:51.385958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.385928    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5hk24"]
Oct 31 16:40:51.489034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.488913    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"]
Oct 31 16:40:51.493364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.493335    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-qxxrw"]
Oct 31 16:40:51.598980 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.598943    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"]
Oct 31 16:40:51.602537 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:51.602512    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-9ms2h"]
Oct 31 16:40:52.439319 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:52.439286    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" path="/var/lib/kubelet/pods/6f39b36f-6f3f-4081-9e40-de9dbff3c623/volumes"
Oct 31 16:40:52.439802 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:52.439786    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" path="/var/lib/kubelet/pods/7a63eb5e-a2ec-4b10-917e-23c984e5d292/volumes"
Oct 31 16:40:52.440202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:52.440190    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" path="/var/lib/kubelet/pods/8350bd2c-9c50-4fb6-bd89-28bab01a2aed/volumes"
Oct 31 16:40:52.440947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:52.440931    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" path="/var/lib/kubelet/pods/e0d6b028-b826-4c73-8625-2c3291adc268/volumes"
Oct 31 16:40:54.004017 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:40:54.003980    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:40:57.634428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:57.634393    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"]
Oct 31 16:40:57.682756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:57.682725    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d04acd-9x27t"]
Oct 31 16:40:58.438495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:40:58.438446    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d388967-7fbc-4da2-83db-ac5ec32909e3" path="/var/lib/kubelet/pods/9d388967-7fbc-4da2-83db-ac5ec32909e3/volumes"
Oct 31 16:41:04.019321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.019288    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"]
Oct 31 16:41:04.019848 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.019589    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy" containerID="cri-o://1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e" gracePeriod=30
Oct 31 16:41:04.839546 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.839511    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"]
Oct 31 16:41:04.840111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840096    4519 cpu_manager.go:414] "containerMap: removing container" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy"
Oct 31 16:41:04.840163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840113    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy"
Oct 31 16:41:04.840163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840124    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f465efc6-db89-4a9e-b02e-5cc199f72782" containerName="revoke-keys"
Oct 31 16:41:04.840163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840130    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f465efc6-db89-4a9e-b02e-5cc199f72782" containerName="revoke-keys"
Oct 31 16:41:04.840163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840145    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy"
Oct 31 16:41:04.840163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840152    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy"
Oct 31 16:41:04.840163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840160    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840166    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840176    4519 cpu_manager.go:414] "containerMap: removing container" podUID="208030bb-9d6a-4b0a-91f6-e81fd8e615b5" containerName="revoke-keys"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840181    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208030bb-9d6a-4b0a-91f6-e81fd8e615b5" containerName="revoke-keys"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840201    4519 cpu_manager.go:414] "containerMap: removing container" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840206    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840223    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840228    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840305    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="8350bd2c-9c50-4fb6-bd89-28bab01a2aed" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840315    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f465efc6-db89-4a9e-b02e-5cc199f72782" containerName="revoke-keys"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840321    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d6b028-b826-4c73-8625-2c3291adc268" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840328    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840333    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="208030bb-9d6a-4b0a-91f6-e81fd8e615b5" containerName="revoke-keys"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840340    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840350    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="667e8d38-e5ab-4af8-a35a-def8a1083241" containerName="main"
Oct 31 16:41:04.840359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.840355    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a63eb5e-a2ec-4b10-917e-23c984e5d292" containerName="istio-proxy"
Oct 31 16:41:04.843917 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.843897    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:04.893935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.893898    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:41:04.894172 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.894147    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:41:04.894275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.894243    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:41:04.894477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.894445    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:41:04.897597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.897575    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"]
Oct 31 16:41:04.933298 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.933265    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:04.933484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.933412    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p45t\" (UniqueName: \"kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:04.933540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:04.933494    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.035268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.035233    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p45t\" (UniqueName: \"kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.035718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.035282    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.035718 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.035372    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.036144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.036122    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.037949 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.037924    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.089097 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.089066    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p45t\" (UniqueName: \"kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.157692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.157612    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:05.525766 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:05.525731    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"]
Oct 31 16:41:05.526023 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:41:05.525998    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7636052_bf94_4a33_b785_7948b76450b6.slice/crio-7487071df52f2b805f0fc6803657ec87ef6df8a8cc65a03a37a2abb78fd9ad1b WatchSource:0}: Error finding container 7487071df52f2b805f0fc6803657ec87ef6df8a8cc65a03a37a2abb78fd9ad1b: Status 404 returned error can't find the container with id 7487071df52f2b805f0fc6803657ec87ef6df8a8cc65a03a37a2abb78fd9ad1b
Oct 31 16:41:06.334524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:06.334483    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252" event={"ID":"c7636052-bf94-4a33-b785-7948b76450b6","Type":"ContainerStarted","Data":"1abc04b566bb96bc591e183aefbc5a14c0f6e5f49a3b91fee4ac484a6915a18d"}
Oct 31 16:41:06.334940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:06.334538    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252" event={"ID":"c7636052-bf94-4a33-b785-7948b76450b6","Type":"ContainerStarted","Data":"7487071df52f2b805f0fc6803657ec87ef6df8a8cc65a03a37a2abb78fd9ad1b"}
Oct 31 16:41:06.444182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:06.444115    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252" podStartSLOduration=2.444097873 podStartE2EDuration="2.444097873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:41:06.441349162 +0000 UTC m=+2310.141278473" watchObservedRunningTime="2026-10-31 16:41:06.444097873 +0000 UTC m=+2310.144027186"
Oct 31 16:41:07.343470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:07.343431    4519 generic.go:356] "Generic (PLEG): container finished" podID="c7636052-bf94-4a33-b785-7948b76450b6" containerID="1abc04b566bb96bc591e183aefbc5a14c0f6e5f49a3b91fee4ac484a6915a18d" exitCode=0
Oct 31 16:41:07.343884 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:07.343518    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252" event={"ID":"c7636052-bf94-4a33-b785-7948b76450b6","Type":"ContainerDied","Data":"1abc04b566bb96bc591e183aefbc5a14c0f6e5f49a3b91fee4ac484a6915a18d"}
Oct 31 16:41:08.490325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.490294    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:08.577153 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.577117    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t\" (UniqueName: \"kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t\") pod \"c7636052-bf94-4a33-b785-7948b76450b6\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") "
Oct 31 16:41:08.577365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.577177    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca\") pod \"c7636052-bf94-4a33-b785-7948b76450b6\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") "
Oct 31 16:41:08.577365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.577201    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca\") pod \"c7636052-bf94-4a33-b785-7948b76450b6\" (UID: \"c7636052-bf94-4a33-b785-7948b76450b6\") "
Oct 31 16:41:08.577580 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.577558    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca" pod "c7636052-bf94-4a33-b785-7948b76450b6" (UID: "c7636052-bf94-4a33-b785-7948b76450b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:08.577736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.577707    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:08.579379 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.579358    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca" pod "c7636052-bf94-4a33-b785-7948b76450b6" (UID: "c7636052-bf94-4a33-b785-7948b76450b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:08.579500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.579380    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t" pod "c7636052-bf94-4a33-b785-7948b76450b6" (UID: "c7636052-bf94-4a33-b785-7948b76450b6"). InnerVolumeSpecName "kube-api-access-2p45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:08.678721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.678642    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7636052-bf94-4a33-b785-7948b76450b6-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:08.678721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:08.678669    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p45t\" (UniqueName: \"kubernetes.io/projected/c7636052-bf94-4a33-b785-7948b76450b6-kube-api-access-2p45t\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.276192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.276164    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:41:09.359046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.358950    4519 generic.go:356] "Generic (PLEG): container finished" podID="23f8058a-6c74-450b-8481-0e5900e34651" containerID="1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e" exitCode=0
Oct 31 16:41:09.359046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.359000    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"
Oct 31 16:41:09.359046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.359007    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" event={"ID":"23f8058a-6c74-450b-8481-0e5900e34651","Type":"ContainerDied","Data":"1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e"}
Oct 31 16:41:09.359335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.359066    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp" event={"ID":"23f8058a-6c74-450b-8481-0e5900e34651","Type":"ContainerDied","Data":"8ff16e7a987ebdc91f7bbd181dfd78b778cc10b0b9db6e1d111f5e840ce9ee40"}
Oct 31 16:41:09.359335 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.359097    4519 scope.go:122] "RemoveContainer" containerID="1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e"
Oct 31 16:41:09.360479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.360442    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"
Oct 31 16:41:09.360582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.360496    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252" event={"ID":"c7636052-bf94-4a33-b785-7948b76450b6","Type":"ContainerDied","Data":"7487071df52f2b805f0fc6803657ec87ef6df8a8cc65a03a37a2abb78fd9ad1b"}
Oct 31 16:41:09.360582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.360538    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7487071df52f2b805f0fc6803657ec87ef6df8a8cc65a03a37a2abb78fd9ad1b"
Oct 31 16:41:09.370983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.370968    4519 scope.go:122] "RemoveContainer" containerID="b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"
Oct 31 16:41:09.380354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.380332    4519 scope.go:122] "RemoveContainer" containerID="1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e"
Oct 31 16:41:09.380723 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:09.380698    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e\": container with ID starting with 1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e not found: ID does not exist" containerID="1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e"
Oct 31 16:41:09.380798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.380732    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e"} err="failed to get container status \"1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e\": rpc error: code = NotFound desc = could not find container \"1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e\": container with ID starting with 1c2848a62b66c63ddcb21755d1abff159e7d14034bf58f8e9757505483d6ed0e not found: ID does not exist"
Oct 31 16:41:09.380798 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.380752    4519 scope.go:122] "RemoveContainer" containerID="b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"
Oct 31 16:41:09.381020 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:09.381001    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def\": container with ID starting with b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def not found: ID does not exist" containerID="b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"
Oct 31 16:41:09.381069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.381027    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def"} err="failed to get container status \"b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def\": rpc error: code = NotFound desc = could not find container \"b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def\": container with ID starting with b89e657490d3d511e0756f2cc53ba61072e7a4150e7e315209e495111fc14def not found: ID does not exist"
Oct 31 16:41:09.387074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387050    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.387495 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387470    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.387618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387592    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.387692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387645    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.387757 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387698    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.387817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387805    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.387897 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.388872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.388872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.387930    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket\") pod \"23f8058a-6c74-450b-8481-0e5900e34651\" (UID: \"23f8058a-6c74-450b-8481-0e5900e34651\") "
Oct 31 16:41:09.388872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.388045    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:09.388872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.388360    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:09.388872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.388745    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:09.389901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.388918    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.389901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.388945    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.389901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.388961    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.389901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.389383    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:09.389901 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.389566    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:09.390966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.390929    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:41:09.391964 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.391941    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:09.392064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.391979    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "kube-api-access-x9l5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:09.392344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.392312    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy" pod "23f8058a-6c74-450b-8481-0e5900e34651" (UID: "23f8058a-6c74-450b-8481-0e5900e34651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:09.489926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.489893    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8058a-6c74-450b-8481-0e5900e34651-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.489926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.489919    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9l5h\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-kube-api-access-x9l5h\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.489926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.489929    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.490258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.489941    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8058a-6c74-450b-8481-0e5900e34651-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.490258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.489951    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8058a-6c74-450b-8481-0e5900e34651-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.490258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.489960    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8058a-6c74-450b-8481-0e5900e34651-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:09.778899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.778859    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"]
Oct 31 16:41:09.786588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:09.784183    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d04acd-openshift-default-6496b7b465-c28vp"]
Oct 31 16:41:10.438759 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:10.438723    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f8058a-6c74-450b-8481-0e5900e34651" path="/var/lib/kubelet/pods/23f8058a-6c74-450b-8481-0e5900e34651/volumes"
Oct 31 16:41:14.664554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:14.664509    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"]
Oct 31 16:41:14.734483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:14.730826    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4fe3a4-k2htw"]
Oct 31 16:41:16.439068 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:16.438957    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f" path="/var/lib/kubelet/pods/cd9e5dc2-f15d-4ba7-85a2-fdace7537c1f/volumes"
Oct 31 16:41:21.452877 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:21.452840    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"]
Oct 31 16:41:21.453299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:21.453093    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy" containerID="cri-o://6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3" gracePeriod=30
Oct 31 16:41:22.839348 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:22.839313    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"]
Oct 31 16:41:22.839816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:22.839567    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" containerID="cri-o://2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017" gracePeriod=30
Oct 31 16:41:24.005183 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:24.005146    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:41:26.210935 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.210893    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"]
Oct 31 16:41:26.257409 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.257377    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-59357a-fn5g7"]
Oct 31 16:41:26.440079 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.440046    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca142586-ff7d-466d-9cd8-2337c1365df1" path="/var/lib/kubelet/pods/ca142586-ff7d-466d-9cd8-2337c1365df1/volumes"
Oct 31 16:41:26.728248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.728189    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:41:26.849348 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849315    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849365    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849395    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849427    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849502    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849545    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849582    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849620    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849676    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98\") pod \"ee9601ce-9efe-4454-8065-20d09ba0056e\" (UID: \"ee9601ce-9efe-4454-8065-20d09ba0056e\") "
Oct 31 16:41:26.849826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.849790    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:26.850041 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850004    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:26.850192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850171    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:26.850284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850258    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:26.850513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850491    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.850602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850522    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.850602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850539    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.850602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850553    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9601ce-9efe-4454-8065-20d09ba0056e-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.850751 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.850630    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:26.851848 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.851817    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:26.851955 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.851892    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:26.852396 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.852370    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "kube-api-access-qfr98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:26.852542 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.852402    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo" pod "ee9601ce-9efe-4454-8065-20d09ba0056e" (UID: "ee9601ce-9efe-4454-8065-20d09ba0056e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:41:26.953017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.952308    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.953017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.952347    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.953017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.952361    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9601ce-9efe-4454-8065-20d09ba0056e-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.953017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.952375    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9601ce-9efe-4454-8065-20d09ba0056e-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:26.953017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:26.952391    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfr98\" (UniqueName: \"kubernetes.io/projected/ee9601ce-9efe-4454-8065-20d09ba0056e-kube-api-access-qfr98\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:27.533863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.533830    4519 generic.go:356] "Generic (PLEG): container finished" podID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerID="6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3" exitCode=0
Oct 31 16:41:27.534315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.533873    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"
Oct 31 16:41:27.534315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.533915    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" event={"ID":"ee9601ce-9efe-4454-8065-20d09ba0056e","Type":"ContainerDied","Data":"6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3"}
Oct 31 16:41:27.534315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.533968    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h" event={"ID":"ee9601ce-9efe-4454-8065-20d09ba0056e","Type":"ContainerDied","Data":"fb81457d39c64384560af5322cfab93d3e4c7dc85b1d24f2b963b7acf2595c2f"}
Oct 31 16:41:27.534315 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.533989    4519 scope.go:122] "RemoveContainer" containerID="6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3"
Oct 31 16:41:27.545842 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.545826    4519 scope.go:122] "RemoveContainer" containerID="6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3"
Oct 31 16:41:27.546125 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:27.546104    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3\": container with ID starting with 6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3 not found: ID does not exist" containerID="6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3"
Oct 31 16:41:27.546199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.546132    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3"} err="failed to get container status \"6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3\": rpc error: code = NotFound desc = could not find container \"6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3\": container with ID starting with 6f43a341f709e1c3f62d0c32c901ebb767645f62333f4d2bba6c458129ff42e3 not found: ID does not exist"
Oct 31 16:41:27.565641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.565613    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:41:27.565749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.565668    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:41:27.691656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.691618    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"]
Oct 31 16:41:27.695538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:27.695513    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-mr56h"]
Oct 31 16:41:28.098484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.098436    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:41:28.268094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268059    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268111    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268154    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268186    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268295 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268268    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268310    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268336    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268386    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268422    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy\") pod \"4531531c-a5f7-45da-ae82-e00f6fc372cd\" (UID: \"4531531c-a5f7-45da-ae82-e00f6fc372cd\") "
Oct 31 16:41:28.268574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268527    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:28.268574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268562    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:28.268850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268698    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:28.268850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268831    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:28.268941 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.268840    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:28.269127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.269102    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4531531c-a5f7-45da-ae82-e00f6fc372cd-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.269224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.269134    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.269224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.269150    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.269224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.269164    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.269224 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.269177    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.270648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.270621    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:28.270648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.270627    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:41:28.270788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.270651    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "kube-api-access-f7vvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:28.270788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.270737    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy" pod "4531531c-a5f7-45da-ae82-e00f6fc372cd" (UID: "4531531c-a5f7-45da-ae82-e00f6fc372cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:28.369858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.369830    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.369858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.369857    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.370075 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.369870    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vvc\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-kube-api-access-f7vvc\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.370075 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.369880    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4531531c-a5f7-45da-ae82-e00f6fc372cd-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:28.438734 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.438701    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" path="/var/lib/kubelet/pods/ee9601ce-9efe-4454-8065-20d09ba0056e/volumes"
Oct 31 16:41:28.542061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.541973    4519 generic.go:356] "Generic (PLEG): container finished" podID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerID="2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017" exitCode=0
Oct 31 16:41:28.542061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.542041    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"
Oct 31 16:41:28.542606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.542072    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" event={"ID":"4531531c-a5f7-45da-ae82-e00f6fc372cd","Type":"ContainerDied","Data":"2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017"}
Oct 31 16:41:28.542606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.542138    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln" event={"ID":"4531531c-a5f7-45da-ae82-e00f6fc372cd","Type":"ContainerDied","Data":"eac94c3a087a719a7f28074157731d1f3c1a7e9e7a675ae44aa183d1b908a611"}
Oct 31 16:41:28.542606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.542170    4519 scope.go:122] "RemoveContainer" containerID="2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017"
Oct 31 16:41:28.552648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.552588    4519 scope.go:122] "RemoveContainer" containerID="2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017"
Oct 31 16:41:28.552859 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:28.552840    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017\": container with ID starting with 2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017 not found: ID does not exist" containerID="2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017"
Oct 31 16:41:28.552913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.552867    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017"} err="failed to get container status \"2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017\": rpc error: code = NotFound desc = could not find container \"2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017\": container with ID starting with 2b8d62a4712f973f0d55f7cf3f0381171c271c5a6287545aef1ffa4ede046017 not found: ID does not exist"
Oct 31 16:41:28.698863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.698829    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"]
Oct 31 16:41:28.746046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:28.746015    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-4bwln"]
Oct 31 16:41:30.438485 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:30.438436    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" path="/var/lib/kubelet/pods/4531531c-a5f7-45da-ae82-e00f6fc372cd/volumes"
Oct 31 16:41:33.019216 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:33.019173    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"]
Oct 31 16:41:33.019644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:33.019421    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy" containerID="cri-o://d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f" gracePeriod=30
Oct 31 16:41:38.304428 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.304367    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:41:38.383013 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.382981    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383031    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383069    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383125    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383148    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383183    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383253    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383280    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383369    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:38.383423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383411    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token\") pod \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\" (UID: \"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f\") "
Oct 31 16:41:38.383743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383542    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:38.383743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383657    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:38.383743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383724    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:38.384005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.383984    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.384005 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.384004    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.384141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.384013    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.384141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.384024    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.384265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.384178    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:38.385297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.385272    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "kube-api-access-q4g2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:38.385498 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.385446    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:38.385662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.385631    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:41:38.385763 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.385693    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token" pod "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" (UID: "f2af1c77-80bd-4ed1-95cd-c35c8ef1080f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:38.485146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.485113    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.485146 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.485148    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4g2c\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-kube-api-access-q4g2c\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.485406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.485166    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.485406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.485176    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.485406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.485184    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:38.618830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.618752    4519 generic.go:356] "Generic (PLEG): container finished" podID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerID="d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f" exitCode=0
Oct 31 16:41:38.618830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.618821    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"
Oct 31 16:41:38.619031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.618839    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" event={"ID":"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f","Type":"ContainerDied","Data":"d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f"}
Oct 31 16:41:38.619031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.618896    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk" event={"ID":"f2af1c77-80bd-4ed1-95cd-c35c8ef1080f","Type":"ContainerDied","Data":"490f7182eacf4c528ed745a8d18cdf43dc4f7cb508a130f8fbaca53616b0ebf3"}
Oct 31 16:41:38.619031 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.618919    4519 scope.go:122] "RemoveContainer" containerID="d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f"
Oct 31 16:41:38.629346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.629330    4519 scope.go:122] "RemoveContainer" containerID="3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"
Oct 31 16:41:38.638180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.638156    4519 scope.go:122] "RemoveContainer" containerID="d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f"
Oct 31 16:41:38.638488 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:38.638441    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f\": container with ID starting with d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f not found: ID does not exist" containerID="d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f"
Oct 31 16:41:38.638594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.638484    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f"} err="failed to get container status \"d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f\": rpc error: code = NotFound desc = could not find container \"d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f\": container with ID starting with d754a8925c55ef649737a8bef3dabe283892372eec52e8f8dc1e93389d40934f not found: ID does not exist"
Oct 31 16:41:38.638594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.638501    4519 scope.go:122] "RemoveContainer" containerID="3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"
Oct 31 16:41:38.638779 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:38.638760    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff\": container with ID starting with 3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff not found: ID does not exist" containerID="3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"
Oct 31 16:41:38.638825 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.638785    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff"} err="failed to get container status \"3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff\": rpc error: code = NotFound desc = could not find container \"3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff\": container with ID starting with 3f6aa694e33177df46a050b66abab08252b1309854a5b989d3ef496a178c5eff not found: ID does not exist"
Oct 31 16:41:38.774600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.774562    4519 scope.go:122] "RemoveContainer" containerID="86aa5647919f1df6eabba0dfc29a65a02879a8c63211a75e4a0922faf8ed1821"
Oct 31 16:41:38.776983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.776962    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"]
Oct 31 16:41:38.779853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.779831    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-gn5jk"]
Oct 31 16:41:38.787965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.787939    4519 scope.go:122] "RemoveContainer" containerID="fe4aa5b39cac1b832ff69c7a02c3a509d51b216643cae8250cce1b528c007529"
Oct 31 16:41:38.797248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.797230    4519 scope.go:122] "RemoveContainer" containerID="1bb8aeef58376a27e72351d4901b4fce1bc43db98a26c277698ad66e68615230"
Oct 31 16:41:38.806808 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.806781    4519 scope.go:122] "RemoveContainer" containerID="68c0c8902830d248fd2f5b5ed91735a95940699c89ed99d6fce674f5767b8a9c"
Oct 31 16:41:38.815404 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:38.815385    4519 scope.go:122] "RemoveContainer" containerID="9b390d78022c325904e3c454e5bc3c0118224f6a8a1d41b6f17e7f09543410b4"
Oct 31 16:41:40.444510 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:40.444476    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" path="/var/lib/kubelet/pods/f2af1c77-80bd-4ed1-95cd-c35c8ef1080f/volumes"
Oct 31 16:41:43.033670 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.033633    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"]
Oct 31 16:41:43.034143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034129    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034144    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034155    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034162    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034170    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c7636052-bf94-4a33-b785-7948b76450b6" containerName="revoke-keys"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034175    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7636052-bf94-4a33-b785-7948b76450b6" containerName="revoke-keys"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034189    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy"
Oct 31 16:41:43.034194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034194    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034206    4519 cpu_manager.go:414] "containerMap: removing container" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034211    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f39b36f-6f3f-4081-9e40-de9dbff3c623" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034224    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034228    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034237    4519 cpu_manager.go:414] "containerMap: removing container" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034241    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034255    4519 cpu_manager.go:414] "containerMap: removing container" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034259    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034329    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034336    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7636052-bf94-4a33-b785-7948b76450b6" containerName="revoke-keys"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034349    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4531531c-a5f7-45da-ae82-e00f6fc372cd" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034356    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034362    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f8058a-6c74-450b-8481-0e5900e34651" containerName="istio-proxy"
Oct 31 16:41:43.034416 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.034369    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9601ce-9efe-4454-8065-20d09ba0056e" containerName="istio-proxy"
Oct 31 16:41:43.039789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.039770    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.090511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.090479    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"]
Oct 31 16:41:43.134323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.134290    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:41:43.134323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.134306    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:41:43.145202 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.145168    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.145352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.145238    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvclp\" (UniqueName: \"kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.145352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.145334    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.178958 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.178936    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:41:43.181682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.181668    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:41:43.246474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.246422    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.246703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.246525    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvclp\" (UniqueName: \"kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.246703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.246574    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.247191 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.247161    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.249072 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.249044    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.300406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.300328    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvclp\" (UniqueName: \"kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp\") pod \"maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.353611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.353576    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:43.724777 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:43.724746    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"]
Oct 31 16:41:43.726070 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:41:43.726040    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9817cb_7b05_4517_93a4_daf63de06c45.slice/crio-8f5fcfbb112a1eed6580da83e6b7842693e381d085f738a6115c27d2ad40226b WatchSource:0}: Error finding container 8f5fcfbb112a1eed6580da83e6b7842693e381d085f738a6115c27d2ad40226b: Status 404 returned error can't find the container with id 8f5fcfbb112a1eed6580da83e6b7842693e381d085f738a6115c27d2ad40226b
Oct 31 16:41:44.661183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:44.661140    4519 generic.go:356] "Generic (PLEG): container finished" podID="dd9817cb-7b05-4517-93a4-daf63de06c45" containerID="8b071132c957e626e6d234efd4a7f99b465884f5cf378ba54d48ac18a4d10a16" exitCode=0
Oct 31 16:41:44.661597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:44.661224    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l" event={"ID":"dd9817cb-7b05-4517-93a4-daf63de06c45","Type":"ContainerDied","Data":"8b071132c957e626e6d234efd4a7f99b465884f5cf378ba54d48ac18a4d10a16"}
Oct 31 16:41:44.661597 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:44.661285    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l" event={"ID":"dd9817cb-7b05-4517-93a4-daf63de06c45","Type":"ContainerStarted","Data":"8f5fcfbb112a1eed6580da83e6b7842693e381d085f738a6115c27d2ad40226b"}
Oct 31 16:41:45.828657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.828629    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:45.878022 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.877993    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca\") pod \"dd9817cb-7b05-4517-93a4-daf63de06c45\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") "
Oct 31 16:41:45.878175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.878041    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca\") pod \"dd9817cb-7b05-4517-93a4-daf63de06c45\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") "
Oct 31 16:41:45.878175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.878105    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp\" (UniqueName: \"kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp\") pod \"dd9817cb-7b05-4517-93a4-daf63de06c45\" (UID: \"dd9817cb-7b05-4517-93a4-daf63de06c45\") "
Oct 31 16:41:45.878442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.878414    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca" pod "dd9817cb-7b05-4517-93a4-daf63de06c45" (UID: "dd9817cb-7b05-4517-93a4-daf63de06c45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:45.878646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.878630    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:45.880177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.880156    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca" pod "dd9817cb-7b05-4517-93a4-daf63de06c45" (UID: "dd9817cb-7b05-4517-93a4-daf63de06c45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:45.880390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.880368    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp" pod "dd9817cb-7b05-4517-93a4-daf63de06c45" (UID: "dd9817cb-7b05-4517-93a4-daf63de06c45"). InnerVolumeSpecName "kube-api-access-lvclp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:45.980015 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.979983    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9817cb-7b05-4517-93a4-daf63de06c45-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:45.980015 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:45.980012    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvclp\" (UniqueName: \"kubernetes.io/projected/dd9817cb-7b05-4517-93a4-daf63de06c45-kube-api-access-lvclp\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:46.676501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:46.676470    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"
Oct 31 16:41:46.676501 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:46.676486    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l" event={"ID":"dd9817cb-7b05-4517-93a4-daf63de06c45","Type":"ContainerDied","Data":"8f5fcfbb112a1eed6580da83e6b7842693e381d085f738a6115c27d2ad40226b"}
Oct 31 16:41:46.676717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:46.676549    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f5fcfbb112a1eed6580da83e6b7842693e381d085f738a6115c27d2ad40226b"
Oct 31 16:41:47.467887 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:47.467854    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"]
Oct 31 16:41:47.515590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:47.515547    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"]
Oct 31 16:41:47.518318 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:47.518291    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"]
Oct 31 16:41:47.520517 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:47.520492    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-f43174-vr252"]
Oct 31 16:41:48.439931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:48.439898    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7636052-bf94-4a33-b785-7948b76450b6" path="/var/lib/kubelet/pods/c7636052-bf94-4a33-b785-7948b76450b6/volumes"
Oct 31 16:41:54.002857 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:54.002818    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:41:54.137398 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:54.137363    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"]
Oct 31 16:41:54.137634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:54.137597    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" containerID="cri-o://ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af" gracePeriod=30
Oct 31 16:41:54.461709 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:54.461681    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:41:54.465890 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:54.465860    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"]
Oct 31 16:41:54.562254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:54.562219    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"]
Oct 31 16:41:55.186218 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.186177    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"]
Oct 31 16:41:55.186960 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.186935    4519 cpu_manager.go:414] "containerMap: removing container" podUID="dd9817cb-7b05-4517-93a4-daf63de06c45" containerName="revoke-keys"
Oct 31 16:41:55.187060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.186961    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9817cb-7b05-4517-93a4-daf63de06c45" containerName="revoke-keys"
Oct 31 16:41:55.187140 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.187127    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9817cb-7b05-4517-93a4-daf63de06c45" containerName="revoke-keys"
Oct 31 16:41:55.187193 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.187153    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2af1c77-80bd-4ed1-95cd-c35c8ef1080f" containerName="istio-proxy"
Oct 31 16:41:55.191909 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.191887    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.241259 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.241234    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:41:55.241417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.241317    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:41:55.241417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.241348    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:41:55.241579 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.241552    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:41:55.242944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.242920    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"]
Oct 31 16:41:55.288029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.287992    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq54j\" (UniqueName: \"kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.288165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.288097    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.288165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.288147    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.389815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.389779    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.389990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.389843    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.389990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.389937    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq54j\" (UniqueName: \"kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.390483 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.390426    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.392158 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.392135    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.443926 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.443858    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq54j\" (UniqueName: \"kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j\") pod \"maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.504957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.504921    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:55.872770 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:55.872739    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"]
Oct 31 16:41:55.874297 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:41:55.874269    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a534710_564e_4582_98cc_40927fb90207.slice/crio-68c31f42b9910920c3f0a05fbfc8bf87c5045593e4abb33d195aab76d9177af1 WatchSource:0}: Error finding container 68c31f42b9910920c3f0a05fbfc8bf87c5045593e4abb33d195aab76d9177af1: Status 404 returned error can't find the container with id 68c31f42b9910920c3f0a05fbfc8bf87c5045593e4abb33d195aab76d9177af1
Oct 31 16:41:56.745251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:56.745214    4519 generic.go:356] "Generic (PLEG): container finished" podID="1a534710-564e-4582-98cc-40927fb90207" containerID="6110b7773f0adafc450cf57913474e7e703535ea1fec62f83014554d66bc07f2" exitCode=0
Oct 31 16:41:56.745692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:56.745297    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn" event={"ID":"1a534710-564e-4582-98cc-40927fb90207","Type":"ContainerDied","Data":"6110b7773f0adafc450cf57913474e7e703535ea1fec62f83014554d66bc07f2"}
Oct 31 16:41:56.745692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:56.745354    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn" event={"ID":"1a534710-564e-4582-98cc-40927fb90207","Type":"ContainerStarted","Data":"68c31f42b9910920c3f0a05fbfc8bf87c5045593e4abb33d195aab76d9177af1"}
Oct 31 16:41:57.025772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:57.025739    4519 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:41:57.025937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:57.025800    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:41:57.900688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:57.900655    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:58.020177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.020103    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca\") pod \"1a534710-564e-4582-98cc-40927fb90207\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") "
Oct 31 16:41:58.020341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.020233    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j\" (UniqueName: \"kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j\") pod \"1a534710-564e-4582-98cc-40927fb90207\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") "
Oct 31 16:41:58.020341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.020262    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca\") pod \"1a534710-564e-4582-98cc-40927fb90207\" (UID: \"1a534710-564e-4582-98cc-40927fb90207\") "
Oct 31 16:41:58.020544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.020517    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca" pod "1a534710-564e-4582-98cc-40927fb90207" (UID: "1a534710-564e-4582-98cc-40927fb90207"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:58.020794 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.020771    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a534710-564e-4582-98cc-40927fb90207-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:58.022415 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.022385    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca" pod "1a534710-564e-4582-98cc-40927fb90207" (UID: "1a534710-564e-4582-98cc-40927fb90207"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:58.022591 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.022571    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j" pod "1a534710-564e-4582-98cc-40927fb90207" (UID: "1a534710-564e-4582-98cc-40927fb90207"). InnerVolumeSpecName "kube-api-access-xq54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:58.121605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.121569    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq54j\" (UniqueName: \"kubernetes.io/projected/1a534710-564e-4582-98cc-40927fb90207-kube-api-access-xq54j\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:58.121605 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.121598    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a534710-564e-4582-98cc-40927fb90207-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:58.759806 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.759777    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"
Oct 31 16:41:58.759993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.759809    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn" event={"ID":"1a534710-564e-4582-98cc-40927fb90207","Type":"ContainerDied","Data":"68c31f42b9910920c3f0a05fbfc8bf87c5045593e4abb33d195aab76d9177af1"}
Oct 31 16:41:58.759993 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:58.759855    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c31f42b9910920c3f0a05fbfc8bf87c5045593e4abb33d195aab76d9177af1"
Oct 31 16:41:59.402430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.402378    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:41:59.466913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.466867    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerName="payload-processing" containerID="cri-o://8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b" gracePeriod=25
Oct 31 16:41:59.539536 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539505    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.539764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539604    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.539764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539633    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.539764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539651    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.539764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539695    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.539764 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539732    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.540127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539826    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.540127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539938    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.540127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539983    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy\") pod \"b5a758d2-3317-4ee9-bb46-39e7c5476698\" (UID: \"b5a758d2-3317-4ee9-bb46-39e7c5476698\") "
Oct 31 16:41:59.540127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.539998    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:59.540127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540023    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:59.540127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540049    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:59.540425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540216    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:59.540425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540225    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:59.540618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540585    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.540618 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540610    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.540782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540624    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a758d2-3317-4ee9-bb46-39e7c5476698-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.540782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540637    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.540782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.540648    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.542108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.542082    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "kube-api-access-jsdkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:59.542234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.542196    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:41:59.542525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.542502    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:59.542525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.542502    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy" pod "b5a758d2-3317-4ee9-bb46-39e7c5476698" (UID: "b5a758d2-3317-4ee9-bb46-39e7c5476698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:41:59.562971 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.562912    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerName="payload-pre-processing" containerID="cri-o://8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769" gracePeriod=25
Oct 31 16:41:59.593378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.593342    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"]
Oct 31 16:41:59.597854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.597820    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"]
Oct 31 16:41:59.641989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.641958    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.641989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.641987    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdkp\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-kube-api-access-jsdkp\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.641989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.641998    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.642239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.642007    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a758d2-3317-4ee9-bb46-39e7c5476698-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:59.767918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.767887    4519 generic.go:356] "Generic (PLEG): container finished" podID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerID="ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af" exitCode=0
Oct 31 16:41:59.768104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.767941    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"
Oct 31 16:41:59.768104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.767969    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" event={"ID":"b5a758d2-3317-4ee9-bb46-39e7c5476698","Type":"ContainerDied","Data":"ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af"}
Oct 31 16:41:59.768104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.768039    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff" event={"ID":"b5a758d2-3317-4ee9-bb46-39e7c5476698","Type":"ContainerDied","Data":"35904d346743c914eeba03f762dbb94ecf3cab51e0a880cbb0ff03d7d67cd429"}
Oct 31 16:41:59.768104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.768065    4519 scope.go:122] "RemoveContainer" containerID="ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af"
Oct 31 16:41:59.778966 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.778950    4519 scope.go:122] "RemoveContainer" containerID="d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"
Oct 31 16:41:59.788374 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.788351    4519 scope.go:122] "RemoveContainer" containerID="ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af"
Oct 31 16:41:59.788704 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:59.788672    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af\": container with ID starting with ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af not found: ID does not exist" containerID="ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af"
Oct 31 16:41:59.788789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.788703    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af"} err="failed to get container status \"ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af\": rpc error: code = NotFound desc = could not find container \"ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af\": container with ID starting with ee3d90081b622438504d1f3fdf2da5ae7ce41a1f7a17b1807fd96fdcf7b468af not found: ID does not exist"
Oct 31 16:41:59.788789 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.788722    4519 scope.go:122] "RemoveContainer" containerID="d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"
Oct 31 16:41:59.789017 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:41:59.788998    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f\": container with ID starting with d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f not found: ID does not exist" containerID="d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"
Oct 31 16:41:59.789070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.789022    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f"} err="failed to get container status \"d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f\": rpc error: code = NotFound desc = could not find container \"d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f\": container with ID starting with d0c5e24e9b00be00c58bc6e8dff23b9ecf3760c62ca6b10a25a6bde1e38da69f not found: ID does not exist"
Oct 31 16:41:59.914248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.914213    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"]
Oct 31 16:41:59.918590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:41:59.918560    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-d8vff"]
Oct 31 16:42:00.438570 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:00.438539    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" path="/var/lib/kubelet/pods/b5a758d2-3317-4ee9-bb46-39e7c5476698/volumes"
Oct 31 16:42:02.488977 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.488925    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main" containerID="cri-o://468c02d75ce77541ca7b250f5555f69023bd98d5f1a93052b884e0925fe6f818" gracePeriod=45
Oct 31 16:42:02.535323 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.535287    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main" containerID="cri-o://6dfd4818f996f225c832dfc2d74b156ac6770734a23c95ef47a180b78c183de7" gracePeriod=45
Oct 31 16:42:02.791344 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.791312    4519 generic.go:356] "Generic (PLEG): container finished" podID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerID="6dfd4818f996f225c832dfc2d74b156ac6770734a23c95ef47a180b78c183de7" exitCode=0
Oct 31 16:42:02.791527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.791368    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" event={"ID":"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a","Type":"ContainerDied","Data":"6dfd4818f996f225c832dfc2d74b156ac6770734a23c95ef47a180b78c183de7"}
Oct 31 16:42:02.792850 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.792829    4519 generic.go:356] "Generic (PLEG): container finished" podID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerID="468c02d75ce77541ca7b250f5555f69023bd98d5f1a93052b884e0925fe6f818" exitCode=0
Oct 31 16:42:02.792954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.792904    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" event={"ID":"bec0b6b0-05b2-42a1-9759-b765b6126855","Type":"ContainerDied","Data":"468c02d75ce77541ca7b250f5555f69023bd98d5f1a93052b884e0925fe6f818"}
Oct 31 16:42:02.792954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.792942    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b" event={"ID":"bec0b6b0-05b2-42a1-9759-b765b6126855","Type":"ContainerDied","Data":"080299d172a1b99c3751c59b7bc0bc8ef60c6e7266268d2ace655618404b1fdc"}
Oct 31 16:42:02.793048 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.792960    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080299d172a1b99c3751c59b7bc0bc8ef60c6e7266268d2ace655618404b1fdc"
Oct 31 16:42:02.804793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.804774    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:42:02.820004 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.819986    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:42:02.978928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.978898    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs\") pod \"bec0b6b0-05b2-42a1-9759-b765b6126855\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") "
Oct 31 16:42:02.979089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.978969    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir\") pod \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") "
Oct 31 16:42:02.979089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.978993    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs\") pod \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") "
Oct 31 16:42:02.979089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979015    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home\") pod \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") "
Oct 31 16:42:02.979089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979062    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm\") pod \"bec0b6b0-05b2-42a1-9759-b765b6126855\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") "
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979091    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache\") pod \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") "
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979121    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir\") pod \"bec0b6b0-05b2-42a1-9759-b765b6126855\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") "
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979157    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm\") pod \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") "
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979228    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home\") pod \"bec0b6b0-05b2-42a1-9759-b765b6126855\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") "
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979264    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache\") pod \"bec0b6b0-05b2-42a1-9759-b765b6126855\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") "
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979274    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir" pod "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" (UID: "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.979302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979295    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home" pod "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" (UID: "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.979712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979401    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc\" (UniqueName: \"kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc\") pod \"bec0b6b0-05b2-42a1-9759-b765b6126855\" (UID: \"bec0b6b0-05b2-42a1-9759-b765b6126855\") "
Oct 31 16:42:02.979712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979435    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg\" (UniqueName: \"kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg\") pod \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\" (UID: \"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a\") "
Oct 31 16:42:02.979712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979539    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir" pod "bec0b6b0-05b2-42a1-9759-b765b6126855" (UID: "bec0b6b0-05b2-42a1-9759-b765b6126855"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.979712 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979599    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home" pod "bec0b6b0-05b2-42a1-9759-b765b6126855" (UID: "bec0b6b0-05b2-42a1-9759-b765b6126855"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.979916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979736    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache" pod "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" (UID: "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.979916 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.979743    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache" pod "bec0b6b0-05b2-42a1-9759-b765b6126855" (UID: "bec0b6b0-05b2-42a1-9759-b765b6126855"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.980071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.980045    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.980071 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.980074    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.980271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.980089    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.980271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.980102    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.980271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.980115    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.980271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.980130    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.981447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.981427    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs" pod "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" (UID: "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:02.981688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.981662    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm" pod "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" (UID: "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.981688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.981672    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs" pod "bec0b6b0-05b2-42a1-9759-b765b6126855" (UID: "bec0b6b0-05b2-42a1-9759-b765b6126855"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:02.981688 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.981679    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm" pod "bec0b6b0-05b2-42a1-9759-b765b6126855" (UID: "bec0b6b0-05b2-42a1-9759-b765b6126855"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.981952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.981930    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc" pod "bec0b6b0-05b2-42a1-9759-b765b6126855" (UID: "bec0b6b0-05b2-42a1-9759-b765b6126855"). InnerVolumeSpecName "kube-api-access-nb7gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:02.982101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:02.982073    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg" pod "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" (UID: "c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a"). InnerVolumeSpecName "kube-api-access-s6wpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:03.081996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.081914    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec0b6b0-05b2-42a1-9759-b765b6126855-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:03.081996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.081948    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:03.081996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.081959    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec0b6b0-05b2-42a1-9759-b765b6126855-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:03.081996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.081968    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:03.081996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.081977    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7gc\" (UniqueName: \"kubernetes.io/projected/bec0b6b0-05b2-42a1-9759-b765b6126855-kube-api-access-nb7gc\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:03.081996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.081987    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wpg\" (UniqueName: \"kubernetes.io/projected/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a-kube-api-access-s6wpg\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:03.760990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.760911    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"]
Oct 31 16:42:03.761427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761413    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761428    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761441    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761446    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761472    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761479    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761500    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main"
Oct 31 16:42:03.761508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761504    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761515    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1a534710-564e-4582-98cc-40927fb90207" containerName="revoke-keys"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761521    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a534710-564e-4582-98cc-40927fb90207" containerName="revoke-keys"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761587    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761596    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a758d2-3317-4ee9-bb46-39e7c5476698" containerName="istio-proxy"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761603    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" containerName="main"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761611    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" containerName="main"
Oct 31 16:42:03.761731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.761618    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a534710-564e-4582-98cc-40927fb90207" containerName="revoke-keys"
Oct 31 16:42:03.775768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.775741    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.800292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.800258    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn" event={"ID":"c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a","Type":"ContainerDied","Data":"5f3a2c54f416c7a4984221db9fb0b66cf1abb2abb1d5c771426b49804c90d227"}
Oct 31 16:42:03.800292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.800286    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"
Oct 31 16:42:03.800523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.800311    4519 scope.go:122] "RemoveContainer" containerID="6dfd4818f996f225c832dfc2d74b156ac6770734a23c95ef47a180b78c183de7"
Oct 31 16:42:03.800523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.800287    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"
Oct 31 16:42:03.817292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.817265    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"]
Oct 31 16:42:03.860843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.860787    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:42:03.860843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.860799    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:42:03.860843 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.860833    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:42:03.873516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.873491    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:42:03.889721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.889688    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.889891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.889764    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkc4t\" (UniqueName: \"kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.889891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.889797    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.989575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.989541    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"]
Oct 31 16:42:03.991569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.991546    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.991635 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.991617    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkc4t\" (UniqueName: \"kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.991724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.991707    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.992212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.992193    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:03.992517 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.992500    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/premium-w0-ac1273-kserve-55d89dc5bf-qzg7b"]
Oct 31 16:42:03.994163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:03.994143    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:04.045274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.045205    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkc4t\" (UniqueName: \"kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t\") pod \"maas-api-revoke-sub-8b9bbbdf-hnxnp\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:04.089208 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.089180    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:04.095205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.095172    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"]
Oct 31 16:42:04.098632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.098611    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/simulator-w0-ac1273-kserve-8c5d4d776-c7jjn"]
Oct 31 16:42:04.438795 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.438719    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec0b6b0-05b2-42a1-9759-b765b6126855" path="/var/lib/kubelet/pods/bec0b6b0-05b2-42a1-9759-b765b6126855/volumes"
Oct 31 16:42:04.439189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.439175    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a" path="/var/lib/kubelet/pods/c6f8a716-8d61-4c9e-af7f-0e1c0bd1572a/volumes"
Oct 31 16:42:04.457243 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.457217    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"]
Oct 31 16:42:04.457843 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:04.457817    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4b23cd_846c_4b6d_b2bd_b89714bed1e5.slice/crio-3504885a1a9e2421fe082eb89f11cdbd6d81e544f0a426331bb1203f1d8086c7 WatchSource:0}: Error finding container 3504885a1a9e2421fe082eb89f11cdbd6d81e544f0a426331bb1203f1d8086c7: Status 404 returned error can't find the container with id 3504885a1a9e2421fe082eb89f11cdbd6d81e544f0a426331bb1203f1d8086c7
Oct 31 16:42:04.809147 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.809107    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp" event={"ID":"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5","Type":"ContainerStarted","Data":"fce3aa35e7414b9bb5671c094103c83e4d386870e365b3ba081c27cf542ce8fc"}
Oct 31 16:42:04.809606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.809175    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp" event={"ID":"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5","Type":"ContainerStarted","Data":"3504885a1a9e2421fe082eb89f11cdbd6d81e544f0a426331bb1203f1d8086c7"}
Oct 31 16:42:04.917707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:04.917654    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp" podStartSLOduration=1.917637562 podStartE2EDuration="1.917637562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:42:04.916131709 +0000 UTC m=+2368.616061020" watchObservedRunningTime="2026-10-31 16:42:04.917637562 +0000 UTC m=+2368.617566917"
Oct 31 16:42:05.819105 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:05.819065    4519 generic.go:356] "Generic (PLEG): container finished" podID="0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" containerID="fce3aa35e7414b9bb5671c094103c83e4d386870e365b3ba081c27cf542ce8fc" exitCode=0
Oct 31 16:42:05.819532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:05.819147    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp" event={"ID":"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5","Type":"ContainerDied","Data":"fce3aa35e7414b9bb5671c094103c83e4d386870e365b3ba081c27cf542ce8fc"}
Oct 31 16:42:06.595782 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:06.595747    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"]
Oct 31 16:42:06.991161 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:06.991119    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:07.024078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.024040    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca\") pod \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") "
Oct 31 16:42:07.024291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.024160    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca\") pod \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") "
Oct 31 16:42:07.024291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.024215    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t\" (UniqueName: \"kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t\") pod \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\" (UID: \"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5\") "
Oct 31 16:42:07.025277 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.025233    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca" pod "0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" (UID: "0a4b23cd-846c-4b6d-b2bd-b89714bed1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:07.028535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.028085    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca" pod "0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" (UID: "0a4b23cd-846c-4b6d-b2bd-b89714bed1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:07.029830 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.029765    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t" pod "0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" (UID: "0a4b23cd-846c-4b6d-b2bd-b89714bed1e5"). InnerVolumeSpecName "kube-api-access-dkc4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:07.128300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.125993    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkc4t\" (UniqueName: \"kubernetes.io/projected/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-kube-api-access-dkc4t\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:07.128300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.126044    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:07.128300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.126070    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:07.834251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.834212    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"
Oct 31 16:42:07.834251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.834240    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp" event={"ID":"0a4b23cd-846c-4b6d-b2bd-b89714bed1e5","Type":"ContainerDied","Data":"3504885a1a9e2421fe082eb89f11cdbd6d81e544f0a426331bb1203f1d8086c7"}
Oct 31 16:42:07.834470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:07.834284    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3504885a1a9e2421fe082eb89f11cdbd6d81e544f0a426331bb1203f1d8086c7"
Oct 31 16:42:08.258448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:08.258406    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:08.258448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:08.258428    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:11.596607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.596560    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing" containerID="cri-o://9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d" gracePeriod=25
Oct 31 16:42:11.643858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.643825    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"]
Oct 31 16:42:11.644563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.644538    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" containerName="revoke-keys"
Oct 31 16:42:11.644563 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.644562    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" containerName="revoke-keys"
Oct 31 16:42:11.644761 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.644747    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" containerName="revoke-keys"
Oct 31 16:42:11.648399 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.648382    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.676532 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.676497    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9nj\" (UniqueName: \"kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.676665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.676617    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.676746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.676670    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.700724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.700700    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"]
Oct 31 16:42:11.704746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.704725    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.744514 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.744481    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:42:11.744725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.744700    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:42:11.744725 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.744729    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:42:11.744928 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.744581    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:42:11.746479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.746443    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"]
Oct 31 16:42:11.756359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.756338    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"]
Oct 31 16:42:11.777984 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.777961    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.778133 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.778014    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.778133 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.778064    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.778133 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.778097    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.778304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.778208    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9nj\" (UniqueName: \"kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.778304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.778270    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9m58\" (UniqueName: \"kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.778730 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.778704    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.780322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.780300    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.832760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.832730    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9nj\" (UniqueName: \"kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj\") pod \"maas-api-revoke-sub-1dbc374d-lqwwf\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.876420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.876353    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"]
Oct 31 16:42:11.878991 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.878967    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.879100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.879004    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.879227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.879204    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9m58\" (UniqueName: \"kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.879833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.879812    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.880588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.880569    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:11.881386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.881367    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.934111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.934082    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"]
Oct 31 16:42:11.937558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.937532    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"]
Oct 31 16:42:11.939752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.939726    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9m58\" (UniqueName: \"kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58\") pod \"maas-api-revoke-sub-04403fee-xwvqr\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:11.942343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.942322    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:11.962612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.962593    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:11.981152 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.981120    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c44k\" (UniqueName: \"kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:11.981271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.981187    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:11.981271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.981254    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:11.981403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.981375    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:11.981482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.981442    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr7fh\" (UniqueName: \"kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:11.981528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.981511    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:11.993163 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:11.993134    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"]
Oct 31 16:42:12.018772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.018742    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:12.083492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.082746    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.083492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.082888    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c44k\" (UniqueName: \"kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.083492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.082945    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.083492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.082991    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.083492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.083070    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.083492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.083123    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr7fh\" (UniqueName: \"kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.085136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.085082    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.085996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.085841    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.087556 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.087507    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.090871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.090789    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.139287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.139222    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c44k\" (UniqueName: \"kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k\") pod \"maas-api-revoke-sub-dcd97a17-zz9zx\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.139551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.139531    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr7fh\" (UniqueName: \"kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh\") pod \"maas-api-revoke-sub-f353f207-nxx4g\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.203595 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.203565    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:12.265394 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.265145    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:12.339924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.339895    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"]
Oct 31 16:42:12.342865 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:12.342830    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e5fcce_5a1e_459e_a82e_07ed0aad66db.slice/crio-3313d11a2b470a988155a5f7668f0d676aacde1a81e6b6553517958b101a765f WatchSource:0}: Error finding container 3313d11a2b470a988155a5f7668f0d676aacde1a81e6b6553517958b101a765f: Status 404 returned error can't find the container with id 3313d11a2b470a988155a5f7668f0d676aacde1a81e6b6553517958b101a765f
Oct 31 16:42:12.398430 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:12.398397    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f45d47_be0b_4e04_aef5_0dbc9639c5f8.slice/crio-8a0b12ecf3259a67b2b7ec8d1d4161c1f8d140d5bc466a3a5ca1184ecf1be261 WatchSource:0}: Error finding container 8a0b12ecf3259a67b2b7ec8d1d4161c1f8d140d5bc466a3a5ca1184ecf1be261: Status 404 returned error can't find the container with id 8a0b12ecf3259a67b2b7ec8d1d4161c1f8d140d5bc466a3a5ca1184ecf1be261
Oct 31 16:42:12.399231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.399205    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"]
Oct 31 16:42:12.568130 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.568094    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"]
Oct 31 16:42:12.568746 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:12.568720    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97177123_adf9_462a_b281_af0a610c259f.slice/crio-6ead78bded66591446f74302d4556e91ecf0e8b4420a2f2edfb6b60828ee3b23 WatchSource:0}: Error finding container 6ead78bded66591446f74302d4556e91ecf0e8b4420a2f2edfb6b60828ee3b23: Status 404 returned error can't find the container with id 6ead78bded66591446f74302d4556e91ecf0e8b4420a2f2edfb6b60828ee3b23
Oct 31 16:42:12.632587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.632562    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"]
Oct 31 16:42:12.668674 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:12.668602    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0a2644_e5c8_4304_a435_54784705a527.slice/crio-fc35d6d6d1a9bedb4208cc033a41993b35fa9371703cb2526b0a7ad9ff684521 WatchSource:0}: Error finding container fc35d6d6d1a9bedb4208cc033a41993b35fa9371703cb2526b0a7ad9ff684521: Status 404 returned error can't find the container with id fc35d6d6d1a9bedb4208cc033a41993b35fa9371703cb2526b0a7ad9ff684521
Oct 31 16:42:12.871081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.871036    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf" event={"ID":"16e5fcce-5a1e-459e-a82e-07ed0aad66db","Type":"ContainerStarted","Data":"1de463b01b98c529ea776f6cd6c5a89ffbfbf192be7257755a8eed8b6abd886a"}
Oct 31 16:42:12.871265 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.871111    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf" event={"ID":"16e5fcce-5a1e-459e-a82e-07ed0aad66db","Type":"ContainerStarted","Data":"3313d11a2b470a988155a5f7668f0d676aacde1a81e6b6553517958b101a765f"}
Oct 31 16:42:12.872708 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.872677    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g" event={"ID":"97177123-adf9-462a-b281-af0a610c259f","Type":"ContainerStarted","Data":"6ead78bded66591446f74302d4556e91ecf0e8b4420a2f2edfb6b60828ee3b23"}
Oct 31 16:42:12.878322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.878295    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr" event={"ID":"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8","Type":"ContainerStarted","Data":"4ea787abd9d86579253bf54e5442a3fd41ba7e2691115f31c7cf18e872f9d307"}
Oct 31 16:42:12.878500 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.878474    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr" event={"ID":"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8","Type":"ContainerStarted","Data":"8a0b12ecf3259a67b2b7ec8d1d4161c1f8d140d5bc466a3a5ca1184ecf1be261"}
Oct 31 16:42:12.879813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.879787    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx" event={"ID":"ec0a2644-e5c8-4304-a435-54784705a527","Type":"ContainerStarted","Data":"fc35d6d6d1a9bedb4208cc033a41993b35fa9371703cb2526b0a7ad9ff684521"}
Oct 31 16:42:12.884138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.884114    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"]
Oct 31 16:42:12.884317 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.884298    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy" containerID="cri-o://dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a" gracePeriod=30
Oct 31 16:42:12.980346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:12.980296    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf" podStartSLOduration=1.980281301 podStartE2EDuration="1.980281301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:42:12.977665402 +0000 UTC m=+2376.677594713" watchObservedRunningTime="2026-10-31 16:42:12.980281301 +0000 UTC m=+2376.680210612"
Oct 31 16:42:13.086025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.085980    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr" podStartSLOduration=2.08596473 podStartE2EDuration="2.08596473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:42:13.084613265 +0000 UTC m=+2376.784542578" watchObservedRunningTime="2026-10-31 16:42:13.08596473 +0000 UTC m=+2376.785894019"
Oct 31 16:42:13.887321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.887239    4519 generic.go:356] "Generic (PLEG): container finished" podID="b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" containerID="4ea787abd9d86579253bf54e5442a3fd41ba7e2691115f31c7cf18e872f9d307" exitCode=0
Oct 31 16:42:13.887827 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.887331    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr" event={"ID":"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8","Type":"ContainerDied","Data":"4ea787abd9d86579253bf54e5442a3fd41ba7e2691115f31c7cf18e872f9d307"}
Oct 31 16:42:13.888974 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.888946    4519 generic.go:356] "Generic (PLEG): container finished" podID="ec0a2644-e5c8-4304-a435-54784705a527" containerID="6a131df04fcfa0f6c603f709ec7b0131ae00a6c410d676f4cf4e6669189fc0c1" exitCode=0
Oct 31 16:42:13.889098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.889008    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx" event={"ID":"ec0a2644-e5c8-4304-a435-54784705a527","Type":"ContainerDied","Data":"6a131df04fcfa0f6c603f709ec7b0131ae00a6c410d676f4cf4e6669189fc0c1"}
Oct 31 16:42:13.890603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.890582    4519 generic.go:356] "Generic (PLEG): container finished" podID="16e5fcce-5a1e-459e-a82e-07ed0aad66db" containerID="1de463b01b98c529ea776f6cd6c5a89ffbfbf192be7257755a8eed8b6abd886a" exitCode=0
Oct 31 16:42:13.890715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.890662    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf" event={"ID":"16e5fcce-5a1e-459e-a82e-07ed0aad66db","Type":"ContainerDied","Data":"1de463b01b98c529ea776f6cd6c5a89ffbfbf192be7257755a8eed8b6abd886a"}
Oct 31 16:42:13.892143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.892124    4519 generic.go:356] "Generic (PLEG): container finished" podID="97177123-adf9-462a-b281-af0a610c259f" containerID="4160542ed820c31f1771625fbca58ad9ec77b34c60cea5583d5c3091b71eabef" exitCode=0
Oct 31 16:42:13.892233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:13.892170    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g" event={"ID":"97177123-adf9-462a-b281-af0a610c259f","Type":"ContainerDied","Data":"4160542ed820c31f1771625fbca58ad9ec77b34c60cea5583d5c3091b71eabef"}
Oct 31 16:42:14.615845 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.615803    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main" containerID="cri-o://b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe" gracePeriod=45
Oct 31 16:42:14.618342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.618308    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main" containerID="cri-o://1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69" gracePeriod=45
Oct 31 16:42:14.897414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.897387    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:42:14.900813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.900791    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:42:14.901059 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.901037    4519 generic.go:356] "Generic (PLEG): container finished" podID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerID="b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe" exitCode=0
Oct 31 16:42:14.901195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.901070    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"
Oct 31 16:42:14.901195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.901125    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" event={"ID":"044ab7fa-9b54-47a0-990f-4b1fb0e049d1","Type":"ContainerDied","Data":"b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe"}
Oct 31 16:42:14.901195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.901166    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t" event={"ID":"044ab7fa-9b54-47a0-990f-4b1fb0e049d1","Type":"ContainerDied","Data":"69694cacaaefe67bee17099e894eef6f99dbfb10c452c09c22cea7be51cf317b"}
Oct 31 16:42:14.901195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.901180    4519 scope.go:122] "RemoveContainer" containerID="b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe"
Oct 31 16:42:14.902716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.902694    4519 generic.go:356] "Generic (PLEG): container finished" podID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerID="1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69" exitCode=0
Oct 31 16:42:14.902803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.902726    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"
Oct 31 16:42:14.902803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.902735    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" event={"ID":"7073ebb0-d92a-4286-bd81-23a6800f9425","Type":"ContainerDied","Data":"1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69"}
Oct 31 16:42:14.902803 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.902774    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6" event={"ID":"7073ebb0-d92a-4286-bd81-23a6800f9425","Type":"ContainerDied","Data":"3c25067d38ff13510e39f7b4210e711259c2fdca5db901f9169d157a7b4daade"}
Oct 31 16:42:14.913262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.913238    4519 scope.go:122] "RemoveContainer" containerID="b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe"
Oct 31 16:42:14.913668 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:14.913637    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe\": container with ID starting with b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe not found: ID does not exist" containerID="b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe"
Oct 31 16:42:14.913775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.913678    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe"} err="failed to get container status \"b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe\": rpc error: code = NotFound desc = could not find container \"b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe\": container with ID starting with b28b839abf9a8ff057fa9c845992b0a5e35422d5aaa5a1342802ffa028617ebe not found: ID does not exist"
Oct 31 16:42:14.913775 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.913704    4519 scope.go:122] "RemoveContainer" containerID="1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69"
Oct 31 16:42:14.926268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.926050    4519 scope.go:122] "RemoveContainer" containerID="1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69"
Oct 31 16:42:14.926893 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:14.926615    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69\": container with ID starting with 1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69 not found: ID does not exist" containerID="1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69"
Oct 31 16:42:14.926893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:14.926655    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69"} err="failed to get container status \"1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69\": rpc error: code = NotFound desc = could not find container \"1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69\": container with ID starting with 1507641aaf7d8a91f24d53e88321be00c3917de52b6e3b4633a748b1619d6c69 not found: ID does not exist"
Oct 31 16:42:15.018044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018011    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home\") pod \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") "
Oct 31 16:42:15.018233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018081    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs\") pod \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") "
Oct 31 16:42:15.018233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018117    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache\") pod \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") "
Oct 31 16:42:15.018233 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018148    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm\") pod \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") "
Oct 31 16:42:15.018419 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018278    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home" pod "044ab7fa-9b54-47a0-990f-4b1fb0e049d1" (UID: "044ab7fa-9b54-47a0-990f-4b1fb0e049d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018543    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache" pod "044ab7fa-9b54-47a0-990f-4b1fb0e049d1" (UID: "044ab7fa-9b54-47a0-990f-4b1fb0e049d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018602    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache\") pod \"7073ebb0-d92a-4286-bd81-23a6800f9425\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018632    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir\") pod \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018679    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2\" (UniqueName: \"kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2\") pod \"7073ebb0-d92a-4286-bd81-23a6800f9425\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018714    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8\" (UniqueName: \"kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8\") pod \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\" (UID: \"044ab7fa-9b54-47a0-990f-4b1fb0e049d1\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018768    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm\") pod \"7073ebb0-d92a-4286-bd81-23a6800f9425\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018797    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home\") pod \"7073ebb0-d92a-4286-bd81-23a6800f9425\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018814    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache" pod "7073ebb0-d92a-4286-bd81-23a6800f9425" (UID: "7073ebb0-d92a-4286-bd81-23a6800f9425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018826    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs\" (UniqueName: \"kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs\") pod \"7073ebb0-d92a-4286-bd81-23a6800f9425\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.018888    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir\") pod \"7073ebb0-d92a-4286-bd81-23a6800f9425\" (UID: \"7073ebb0-d92a-4286-bd81-23a6800f9425\") "
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.019270    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.019284    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.019293    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.019449    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir" pod "7073ebb0-d92a-4286-bd81-23a6800f9425" (UID: "7073ebb0-d92a-4286-bd81-23a6800f9425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.019879    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir" pod "044ab7fa-9b54-47a0-990f-4b1fb0e049d1" (UID: "044ab7fa-9b54-47a0-990f-4b1fb0e049d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.020220 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.020037    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home" pod "7073ebb0-d92a-4286-bd81-23a6800f9425" (UID: "7073ebb0-d92a-4286-bd81-23a6800f9425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.024109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.024086    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs" pod "7073ebb0-d92a-4286-bd81-23a6800f9425" (UID: "7073ebb0-d92a-4286-bd81-23a6800f9425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.026919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.026857    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm" pod "7073ebb0-d92a-4286-bd81-23a6800f9425" (UID: "7073ebb0-d92a-4286-bd81-23a6800f9425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.026919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.026883    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8" pod "044ab7fa-9b54-47a0-990f-4b1fb0e049d1" (UID: "044ab7fa-9b54-47a0-990f-4b1fb0e049d1"). InnerVolumeSpecName "kube-api-access-5ztp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.027103 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.026929    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs" pod "044ab7fa-9b54-47a0-990f-4b1fb0e049d1" (UID: "044ab7fa-9b54-47a0-990f-4b1fb0e049d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.027354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.027297    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2" pod "7073ebb0-d92a-4286-bd81-23a6800f9425" (UID: "7073ebb0-d92a-4286-bd81-23a6800f9425"). InnerVolumeSpecName "kube-api-access-4r5z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.028568 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.028545    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm" pod "044ab7fa-9b54-47a0-990f-4b1fb0e049d1" (UID: "044ab7fa-9b54-47a0-990f-4b1fb0e049d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.059252 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.059226    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:15.109228 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.109198    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:15.121165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121094    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58\" (UniqueName: \"kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58\") pod \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") "
Oct 31 16:42:15.121353 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121278    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca\") pod \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") "
Oct 31 16:42:15.121418 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121334    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca\") pod \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\" (UID: \"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8\") "
Oct 31 16:42:15.121857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121812    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.121857 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121845    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5z2\" (UniqueName: \"kubernetes.io/projected/7073ebb0-d92a-4286-bd81-23a6800f9425-kube-api-access-4r5z2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121861    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ztp8\" (UniqueName: \"kubernetes.io/projected/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-kube-api-access-5ztp8\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121876    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121876    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca" pod "b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" (UID: "b2f45d47-be0b-4e04-aef5-0dbc9639c5f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121890    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121932    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7073ebb0-d92a-4286-bd81-23a6800f9425-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121946    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7073ebb0-d92a-4286-bd81-23a6800f9425-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121960    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.121972    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044ab7fa-9b54-47a0-990f-4b1fb0e049d1-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.122747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.122195    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:15.124212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.124174    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58" pod "b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" (UID: "b2f45d47-be0b-4e04-aef5-0dbc9639c5f8"). InnerVolumeSpecName "kube-api-access-r9m58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.124413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.124393    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca" pod "b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" (UID: "b2f45d47-be0b-4e04-aef5-0dbc9639c5f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.148969 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.148945    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:15.222940 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.222912    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca\") pod \"ec0a2644-e5c8-4304-a435-54784705a527\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") "
Oct 31 16:42:15.223098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.222953    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k\" (UniqueName: \"kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k\") pod \"ec0a2644-e5c8-4304-a435-54784705a527\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") "
Oct 31 16:42:15.223098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223004    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca\") pod \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") "
Oct 31 16:42:15.223098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223034    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca\") pod \"ec0a2644-e5c8-4304-a435-54784705a527\" (UID: \"ec0a2644-e5c8-4304-a435-54784705a527\") "
Oct 31 16:42:15.223098 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223067    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca\") pod \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") "
Oct 31 16:42:15.223321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223158    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj\" (UniqueName: \"kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj\") pod \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\" (UID: \"16e5fcce-5a1e-459e-a82e-07ed0aad66db\") "
Oct 31 16:42:15.223321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223201    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh\" (UniqueName: \"kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh\") pod \"97177123-adf9-462a-b281-af0a610c259f\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") "
Oct 31 16:42:15.223321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223236    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca\") pod \"97177123-adf9-462a-b281-af0a610c259f\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") "
Oct 31 16:42:15.223321 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223271    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca" pod "ec0a2644-e5c8-4304-a435-54784705a527" (UID: "ec0a2644-e5c8-4304-a435-54784705a527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:15.223587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223345    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca\") pod \"97177123-adf9-462a-b281-af0a610c259f\" (UID: \"97177123-adf9-462a-b281-af0a610c259f\") "
Oct 31 16:42:15.223587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223424    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca" pod "16e5fcce-5a1e-459e-a82e-07ed0aad66db" (UID: "16e5fcce-5a1e-459e-a82e-07ed0aad66db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:15.223784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223764    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.223856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223791    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9m58\" (UniqueName: \"kubernetes.io/projected/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-kube-api-access-r9m58\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.223856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223808    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.223856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223822    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.223856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.223838    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.224234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.224172    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca" pod "97177123-adf9-462a-b281-af0a610c259f" (UID: "97177123-adf9-462a-b281-af0a610c259f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:15.225657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.225630    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca" pod "97177123-adf9-462a-b281-af0a610c259f" (UID: "97177123-adf9-462a-b281-af0a610c259f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.226081 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.226038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca" pod "16e5fcce-5a1e-459e-a82e-07ed0aad66db" (UID: "16e5fcce-5a1e-459e-a82e-07ed0aad66db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.226502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.226439    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca" pod "ec0a2644-e5c8-4304-a435-54784705a527" (UID: "ec0a2644-e5c8-4304-a435-54784705a527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.226599 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.226578    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k" pod "ec0a2644-e5c8-4304-a435-54784705a527" (UID: "ec0a2644-e5c8-4304-a435-54784705a527"). InnerVolumeSpecName "kube-api-access-4c44k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.226667 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.226621    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh" pod "97177123-adf9-462a-b281-af0a610c259f" (UID: "97177123-adf9-462a-b281-af0a610c259f"). InnerVolumeSpecName "kube-api-access-sr7fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.226860 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.226845    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj" pod "16e5fcce-5a1e-459e-a82e-07ed0aad66db" (UID: "16e5fcce-5a1e-459e-a82e-07ed0aad66db"). InnerVolumeSpecName "kube-api-access-fg9nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.325232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325194    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97177123-adf9-462a-b281-af0a610c259f-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.325232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325226    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c44k\" (UniqueName: \"kubernetes.io/projected/ec0a2644-e5c8-4304-a435-54784705a527-kube-api-access-4c44k\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.325232 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325237    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0a2644-e5c8-4304-a435-54784705a527-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.325492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325247    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fcce-5a1e-459e-a82e-07ed0aad66db-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.325492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325257    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9nj\" (UniqueName: \"kubernetes.io/projected/16e5fcce-5a1e-459e-a82e-07ed0aad66db-kube-api-access-fg9nj\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.325492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325266    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr7fh\" (UniqueName: \"kubernetes.io/projected/97177123-adf9-462a-b281-af0a610c259f-kube-api-access-sr7fh\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.325492 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.325275    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97177123-adf9-462a-b281-af0a610c259f-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.713471 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.713426    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"]
Oct 31 16:42:15.717254 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.717227    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-w5-ef6d93-kserve-67889bcb57-r4x9t"]
Oct 31 16:42:15.820942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.820907    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"]
Oct 31 16:42:15.823589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.823564    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/premium-w5-ef6d93-kserve-7c9c4645d-74nm6"]
Oct 31 16:42:15.911813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.911736    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"
Oct 31 16:42:15.911813 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.911773    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf" event={"ID":"16e5fcce-5a1e-459e-a82e-07ed0aad66db","Type":"ContainerDied","Data":"3313d11a2b470a988155a5f7668f0d676aacde1a81e6b6553517958b101a765f"}
Oct 31 16:42:15.912284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.911816    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3313d11a2b470a988155a5f7668f0d676aacde1a81e6b6553517958b101a765f"
Oct 31 16:42:15.914866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.914845    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"
Oct 31 16:42:15.914996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.914888    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g" event={"ID":"97177123-adf9-462a-b281-af0a610c259f","Type":"ContainerDied","Data":"6ead78bded66591446f74302d4556e91ecf0e8b4420a2f2edfb6b60828ee3b23"}
Oct 31 16:42:15.914996 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.914934    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ead78bded66591446f74302d4556e91ecf0e8b4420a2f2edfb6b60828ee3b23"
Oct 31 16:42:15.916250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.916228    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"
Oct 31 16:42:15.916387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.916256    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr" event={"ID":"b2f45d47-be0b-4e04-aef5-0dbc9639c5f8","Type":"ContainerDied","Data":"8a0b12ecf3259a67b2b7ec8d1d4161c1f8d140d5bc466a3a5ca1184ecf1be261"}
Oct 31 16:42:15.916387 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.916297    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0b12ecf3259a67b2b7ec8d1d4161c1f8d140d5bc466a3a5ca1184ecf1be261"
Oct 31 16:42:15.917627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.917598    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx" event={"ID":"ec0a2644-e5c8-4304-a435-54784705a527","Type":"ContainerDied","Data":"fc35d6d6d1a9bedb4208cc033a41993b35fa9371703cb2526b0a7ad9ff684521"}
Oct 31 16:42:15.917627 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.917624    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"
Oct 31 16:42:15.917776 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:15.917647    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc35d6d6d1a9bedb4208cc033a41993b35fa9371703cb2526b0a7ad9ff684521"
Oct 31 16:42:16.439034 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:16.438966    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" path="/var/lib/kubelet/pods/044ab7fa-9b54-47a0-990f-4b1fb0e049d1/volumes"
Oct 31 16:42:16.439386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:16.439370    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" path="/var/lib/kubelet/pods/7073ebb0-d92a-4286-bd81-23a6800f9425/volumes"
Oct 31 16:42:18.162541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.162447    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:42:18.254285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254253    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254330    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254437    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254511 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254503    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254542    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254615    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254636    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:18.254700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254689    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254722    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.254908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254782    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:18.254908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254791    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token\") pod \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\" (UID: \"b4e3ea94-9d0a-486f-a60a-29e4ff843650\") "
Oct 31 16:42:18.255053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254882    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:18.255053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.254945    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:18.255235 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.255218    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.255312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.255241    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.255312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.255249    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:18.255312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.255255    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.255312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.255298    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.257157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.257098    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:18.257157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.257118    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "kube-api-access-pvscv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:18.257157 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.257133    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:18.257406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.257343    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo" pod "b4e3ea94-9d0a-486f-a60a-29e4ff843650" (UID: "b4e3ea94-9d0a-486f-a60a-29e4ff843650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:42:18.257889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.257863    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:18.257889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.257876    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:18.356600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.356563    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.356600 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.356602    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvscv\" (UniqueName: \"kubernetes.io/projected/b4e3ea94-9d0a-486f-a60a-29e4ff843650-kube-api-access-pvscv\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.356838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.356620    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.356838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.356634    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.356838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.356648    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e3ea94-9d0a-486f-a60a-29e4ff843650-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.939060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.939027    4519 generic.go:356] "Generic (PLEG): container finished" podID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerID="dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a" exitCode=0
Oct 31 16:42:18.939225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.939078    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"
Oct 31 16:42:18.939225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.939115    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" event={"ID":"b4e3ea94-9d0a-486f-a60a-29e4ff843650","Type":"ContainerDied","Data":"dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a"}
Oct 31 16:42:18.939225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.939169    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj" event={"ID":"b4e3ea94-9d0a-486f-a60a-29e4ff843650","Type":"ContainerDied","Data":"ecb57578d694ad32b67ab143ebcef4354db467a5a51b58e0d0c391c700bf7d9b"}
Oct 31 16:42:18.939225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.939197    4519 scope.go:122] "RemoveContainer" containerID="dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a"
Oct 31 16:42:18.948983 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.948960    4519 scope.go:122] "RemoveContainer" containerID="dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a"
Oct 31 16:42:18.949264 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:18.949236    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a\": container with ID starting with dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a not found: ID does not exist" containerID="dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a"
Oct 31 16:42:18.949337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:18.949271    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a"} err="failed to get container status \"dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a\": rpc error: code = NotFound desc = could not find container \"dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a\": container with ID starting with dd8f162be1354c40c5a2a3755fa7b39d0a031683bb7cc363be8257d562d95b3a not found: ID does not exist"
Oct 31 16:42:19.095816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.095781    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"]
Oct 31 16:42:19.147540 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.144409    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-9qxgj"]
Oct 31 16:42:19.344529 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.344492    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:19.443060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.443029    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"]
Oct 31 16:42:19.735603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.735573    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:42:19.849810 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.849775    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:42:19.874692 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.874663    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume\") pod \"29b3689d-353c-4ce9-99cd-f19df1dfceff\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") "
Oct 31 16:42:19.874900 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.874879    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl\" (UniqueName: \"kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl\") pod \"29b3689d-353c-4ce9-99cd-f19df1dfceff\" (UID: \"29b3689d-353c-4ce9-99cd-f19df1dfceff\") "
Oct 31 16:42:19.875059 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.875036    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume" pod "29b3689d-353c-4ce9-99cd-f19df1dfceff" (UID: "29b3689d-353c-4ce9-99cd-f19df1dfceff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:19.875266 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.875248    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b3689d-353c-4ce9-99cd-f19df1dfceff-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:19.876858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.876838    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl" pod "29b3689d-353c-4ce9-99cd-f19df1dfceff" (UID: "29b3689d-353c-4ce9-99cd-f19df1dfceff"). InnerVolumeSpecName "kube-api-access-c2hhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:19.948366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.948293    4519 generic.go:356] "Generic (PLEG): container finished" podID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerID="8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769" exitCode=0
Oct 31 16:42:19.948366 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.948338    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"
Oct 31 16:42:19.948584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.948372    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" event={"ID":"d236a06f-6b03-401c-97d6-1efae5cc2406","Type":"ContainerDied","Data":"8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769"}
Oct 31 16:42:19.948584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.948443    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q" event={"ID":"d236a06f-6b03-401c-97d6-1efae5cc2406","Type":"ContainerDied","Data":"c5112c384e7e223729576afcabd4cad915223651a3e3fa0e2da78f6b1247ee8b"}
Oct 31 16:42:19.948584 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.948478    4519 scope.go:122] "RemoveContainer" containerID="8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769"
Oct 31 16:42:19.949701 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.949682    4519 generic.go:356] "Generic (PLEG): container finished" podID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerID="8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b" exitCode=0
Oct 31 16:42:19.949784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.949733    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"
Oct 31 16:42:19.949784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.949735    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" event={"ID":"29b3689d-353c-4ce9-99cd-f19df1dfceff","Type":"ContainerDied","Data":"8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b"}
Oct 31 16:42:19.949784 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.949774    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66" event={"ID":"29b3689d-353c-4ce9-99cd-f19df1dfceff","Type":"ContainerDied","Data":"b3748dfc5f8029e04102430d9aa07068eddbfb977845734cdcedbd6ddc33def9"}
Oct 31 16:42:19.960678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.960661    4519 scope.go:122] "RemoveContainer" containerID="8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769"
Oct 31 16:42:19.960962 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:19.960936    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769\": container with ID starting with 8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769 not found: ID does not exist" containerID="8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769"
Oct 31 16:42:19.961055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.960968    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769"} err="failed to get container status \"8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769\": rpc error: code = NotFound desc = could not find container \"8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769\": container with ID starting with 8c146d265029a5e972fabf35af55b2e8b4cc173582ff46aae157b450e807e769 not found: ID does not exist"
Oct 31 16:42:19.961055 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.960987    4519 scope.go:122] "RemoveContainer" containerID="8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b"
Oct 31 16:42:19.970128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.970108    4519 scope.go:122] "RemoveContainer" containerID="8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b"
Oct 31 16:42:19.970440 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:19.970418    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b\": container with ID starting with 8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b not found: ID does not exist" containerID="8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b"
Oct 31 16:42:19.970574 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.970445    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b"} err="failed to get container status \"8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b\": rpc error: code = NotFound desc = could not find container \"8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b\": container with ID starting with 8b0d20c45cdf32cf0adf660e24f0d353b3f209e1ab33b37b09c2eb412e55a19b not found: ID does not exist"
Oct 31 16:42:19.975944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.975912    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76\" (UniqueName: \"kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76\") pod \"d236a06f-6b03-401c-97d6-1efae5cc2406\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") "
Oct 31 16:42:19.976037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.975969    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume\") pod \"d236a06f-6b03-401c-97d6-1efae5cc2406\" (UID: \"d236a06f-6b03-401c-97d6-1efae5cc2406\") "
Oct 31 16:42:19.976311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.976288    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume" pod "d236a06f-6b03-401c-97d6-1efae5cc2406" (UID: "d236a06f-6b03-401c-97d6-1efae5cc2406"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:19.976427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.976395    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d236a06f-6b03-401c-97d6-1efae5cc2406-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:19.976427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.976409    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hhl\" (UniqueName: \"kubernetes.io/projected/29b3689d-353c-4ce9-99cd-f19df1dfceff-kube-api-access-c2hhl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:19.977856 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:19.977835    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76" pod "d236a06f-6b03-401c-97d6-1efae5cc2406" (UID: "d236a06f-6b03-401c-97d6-1efae5cc2406"). InnerVolumeSpecName "kube-api-access-svt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:20.077674 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.077643    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svt76\" (UniqueName: \"kubernetes.io/projected/d236a06f-6b03-401c-97d6-1efae5cc2406-kube-api-access-svt76\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:20.160300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.160263    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"]
Oct 31 16:42:20.163094 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.163068    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ac1273-74fc6446bf-ccs66"]
Oct 31 16:42:20.363543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.363508    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"]
Oct 31 16:42:20.366590 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.366565    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ac1273-787cd546f5-9gg7q"]
Oct 31 16:42:20.438234 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.438199    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" path="/var/lib/kubelet/pods/29b3689d-353c-4ce9-99cd-f19df1dfceff/volumes"
Oct 31 16:42:20.438687 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.438672    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" path="/var/lib/kubelet/pods/b4e3ea94-9d0a-486f-a60a-29e4ff843650/volumes"
Oct 31 16:42:20.439020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:20.439008    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" path="/var/lib/kubelet/pods/d236a06f-6b03-401c-97d6-1efae5cc2406/volumes"
Oct 31 16:42:24.011148 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:24.011109    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:42:28.172903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.172870    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"]
Oct 31 16:42:28.173342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173329    4519 cpu_manager.go:414] "containerMap: removing container" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173343    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173355    4519 cpu_manager.go:414] "containerMap: removing container" podUID="16e5fcce-5a1e-459e-a82e-07ed0aad66db" containerName="revoke-keys"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173360    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e5fcce-5a1e-459e-a82e-07ed0aad66db" containerName="revoke-keys"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173371    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ec0a2644-e5c8-4304-a435-54784705a527" containerName="revoke-keys"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173377    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0a2644-e5c8-4304-a435-54784705a527" containerName="revoke-keys"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173386    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" containerName="revoke-keys"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173391    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" containerName="revoke-keys"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173400    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerName="payload-pre-processing"
Oct 31 16:42:28.173407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173405    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerName="payload-pre-processing"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173416    4519 cpu_manager.go:414] "containerMap: removing container" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173421    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173428    4519 cpu_manager.go:414] "containerMap: removing container" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerName="payload-processing"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173433    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerName="payload-processing"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173442    4519 cpu_manager.go:414] "containerMap: removing container" podUID="97177123-adf9-462a-b281-af0a610c259f" containerName="revoke-keys"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173447    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97177123-adf9-462a-b281-af0a610c259f" containerName="revoke-keys"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173467    4519 cpu_manager.go:414] "containerMap: removing container" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173472    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173537    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="044ab7fa-9b54-47a0-990f-4b1fb0e049d1" containerName="main"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173547    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e5fcce-5a1e-459e-a82e-07ed0aad66db" containerName="revoke-keys"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173555    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0a2644-e5c8-4304-a435-54784705a527" containerName="revoke-keys"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173561    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" containerName="revoke-keys"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173567    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d236a06f-6b03-401c-97d6-1efae5cc2406" containerName="payload-pre-processing"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173573    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="7073ebb0-d92a-4286-bd81-23a6800f9425" containerName="main"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173579    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b3689d-353c-4ce9-99cd-f19df1dfceff" containerName="payload-processing"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173585    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="97177123-adf9-462a-b281-af0a610c259f" containerName="revoke-keys"
Oct 31 16:42:28.173728 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.173590    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e3ea94-9d0a-486f-a60a-29e4ff843650" containerName="istio-proxy"
Oct 31 16:42:28.178524 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.178503    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.230249 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.230218    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"]
Oct 31 16:42:28.257956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.257926    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.257956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.257959    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.258212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.257992    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78nr\" (UniqueName: \"kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.273880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.273831    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:42:28.274010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.273940    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:42:28.274010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.273996    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:42:28.274118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.274015    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:42:28.278662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.278642    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"]
Oct 31 16:42:28.282429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.282415    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.332828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.332804    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"]
Oct 31 16:42:28.359189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.359159    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78nr\" (UniqueName: \"kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.359343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.359211    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.359343 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.359293    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.359499 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.359399    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdb5\" (UniqueName: \"kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.359602 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.359584    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.359663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.359623    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.360174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.360155    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.361752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.361732    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.412844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.412819    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78nr\" (UniqueName: \"kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr\") pod \"maas-api-revoke-sub-1d5948d8-88nzg\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.461037 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.461014    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.461170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.461096    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.461170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.461116    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdb5\" (UniqueName: \"kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.461822 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.461796    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.463238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.463212    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.491438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.491417    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:28.515430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.515406    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdb5\" (UniqueName: \"kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5\") pod \"maas-api-revoke-sub-55df6fe1-pphs5\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.595078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.595049    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:28.859885 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.859847    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"]
Oct 31 16:42:28.860219 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:28.860199    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5596b388_86a8_4ba7_86d2_11f50a01d63f.slice/crio-bc41147600cb4c4e4138c061184ec4368edf6e5c7549ab1f77b5257dda77709f WatchSource:0}: Error finding container bc41147600cb4c4e4138c061184ec4368edf6e5c7549ab1f77b5257dda77709f: Status 404 returned error can't find the container with id bc41147600cb4c4e4138c061184ec4368edf6e5c7549ab1f77b5257dda77709f
Oct 31 16:42:28.964029 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:28.964000    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"]
Oct 31 16:42:28.965097 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:42:28.965060    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449026bc_7a74_4640_b65f_564a85c04705.slice/crio-898a7f4a76c14acb3a853c3c08498ad4634e2a8f36c142ac4741e573e092ad27 WatchSource:0}: Error finding container 898a7f4a76c14acb3a853c3c08498ad4634e2a8f36c142ac4741e573e092ad27: Status 404 returned error can't find the container with id 898a7f4a76c14acb3a853c3c08498ad4634e2a8f36c142ac4741e573e092ad27
Oct 31 16:42:29.017772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:29.017744    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5" event={"ID":"449026bc-7a74-4640-b65f-564a85c04705","Type":"ContainerStarted","Data":"898a7f4a76c14acb3a853c3c08498ad4634e2a8f36c142ac4741e573e092ad27"}
Oct 31 16:42:29.018947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:29.018921    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg" event={"ID":"5596b388-86a8-4ba7-86d2-11f50a01d63f","Type":"ContainerStarted","Data":"bc41147600cb4c4e4138c061184ec4368edf6e5c7549ab1f77b5257dda77709f"}
Oct 31 16:42:29.345262 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:29.345222    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:30.027927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:30.027891    4519 generic.go:356] "Generic (PLEG): container finished" podID="449026bc-7a74-4640-b65f-564a85c04705" containerID="57b9643bc1a0ece95657e9bbd8acd9a600206ba145f031d6c05f41adc5f85496" exitCode=0
Oct 31 16:42:30.028091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:30.027931    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5" event={"ID":"449026bc-7a74-4640-b65f-564a85c04705","Type":"ContainerDied","Data":"57b9643bc1a0ece95657e9bbd8acd9a600206ba145f031d6c05f41adc5f85496"}
Oct 31 16:42:30.029544 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:30.029521    4519 generic.go:356] "Generic (PLEG): container finished" podID="5596b388-86a8-4ba7-86d2-11f50a01d63f" containerID="27088589cc4cd21257e7add52497e0512083141ecffb4a18cce5280a66bf8222" exitCode=0
Oct 31 16:42:30.029662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:30.029582    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg" event={"ID":"5596b388-86a8-4ba7-86d2-11f50a01d63f","Type":"ContainerDied","Data":"27088589cc4cd21257e7add52497e0512083141ecffb4a18cce5280a66bf8222"}
Oct 31 16:42:31.212039 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.212012    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:31.214053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.214030    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:31.310947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.310759    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5\" (UniqueName: \"kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5\") pod \"449026bc-7a74-4640-b65f-564a85c04705\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") "
Oct 31 16:42:31.310947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.310824    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca\") pod \"5596b388-86a8-4ba7-86d2-11f50a01d63f\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") "
Oct 31 16:42:31.310947 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.310882    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr\" (UniqueName: \"kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr\") pod \"5596b388-86a8-4ba7-86d2-11f50a01d63f\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") "
Oct 31 16:42:31.311271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.311027    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca\") pod \"449026bc-7a74-4640-b65f-564a85c04705\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") "
Oct 31 16:42:31.311271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.311094    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca\") pod \"5596b388-86a8-4ba7-86d2-11f50a01d63f\" (UID: \"5596b388-86a8-4ba7-86d2-11f50a01d63f\") "
Oct 31 16:42:31.311271 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.311123    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca\") pod \"449026bc-7a74-4640-b65f-564a85c04705\" (UID: \"449026bc-7a74-4640-b65f-564a85c04705\") "
Oct 31 16:42:31.312562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.312517    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca" pod "5596b388-86a8-4ba7-86d2-11f50a01d63f" (UID: "5596b388-86a8-4ba7-86d2-11f50a01d63f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:31.312750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.312676    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca" pod "449026bc-7a74-4640-b65f-564a85c04705" (UID: "449026bc-7a74-4640-b65f-564a85c04705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:31.316061 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.316032    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca" pod "449026bc-7a74-4640-b65f-564a85c04705" (UID: "449026bc-7a74-4640-b65f-564a85c04705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:31.316214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.316114    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5" pod "449026bc-7a74-4640-b65f-564a85c04705" (UID: "449026bc-7a74-4640-b65f-564a85c04705"). InnerVolumeSpecName "kube-api-access-wzdb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:31.316287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.316267    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr" pod "5596b388-86a8-4ba7-86d2-11f50a01d63f" (UID: "5596b388-86a8-4ba7-86d2-11f50a01d63f"). InnerVolumeSpecName "kube-api-access-m78nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:31.316675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.316649    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca" pod "5596b388-86a8-4ba7-86d2-11f50a01d63f" (UID: "5596b388-86a8-4ba7-86d2-11f50a01d63f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:31.412531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.412492    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/449026bc-7a74-4640-b65f-564a85c04705-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.412531 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.412530    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.412740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.412545    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/449026bc-7a74-4640-b65f-564a85c04705-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.412740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.412560    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdb5\" (UniqueName: \"kubernetes.io/projected/449026bc-7a74-4640-b65f-564a85c04705-kube-api-access-wzdb5\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.412740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.412574    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5596b388-86a8-4ba7-86d2-11f50a01d63f-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.412740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.412586    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78nr\" (UniqueName: \"kubernetes.io/projected/5596b388-86a8-4ba7-86d2-11f50a01d63f-kube-api-access-m78nr\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.831923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:31.831898    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:42:32.018957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.018924    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume\") pod \"4a555145-7546-430a-90c6-a4a96b4a5106\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") "
Oct 31 16:42:32.019131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.019090    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc\" (UniqueName: \"kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc\") pod \"4a555145-7546-430a-90c6-a4a96b4a5106\" (UID: \"4a555145-7546-430a-90c6-a4a96b4a5106\") "
Oct 31 16:42:32.019304 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.019280    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume" pod "4a555145-7546-430a-90c6-a4a96b4a5106" (UID: "4a555145-7546-430a-90c6-a4a96b4a5106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:32.019393 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.019381    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a555145-7546-430a-90c6-a4a96b4a5106-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:32.021245 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.021224    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc" pod "4a555145-7546-430a-90c6-a4a96b4a5106" (UID: "4a555145-7546-430a-90c6-a4a96b4a5106"). InnerVolumeSpecName "kube-api-access-9hwgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:32.046197 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.046175    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"
Oct 31 16:42:32.046325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.046212    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5" event={"ID":"449026bc-7a74-4640-b65f-564a85c04705","Type":"ContainerDied","Data":"898a7f4a76c14acb3a853c3c08498ad4634e2a8f36c142ac4741e573e092ad27"}
Oct 31 16:42:32.046325 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.046257    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898a7f4a76c14acb3a853c3c08498ad4634e2a8f36c142ac4741e573e092ad27"
Oct 31 16:42:32.047711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.047695    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"
Oct 31 16:42:32.047836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.047733    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg" event={"ID":"5596b388-86a8-4ba7-86d2-11f50a01d63f","Type":"ContainerDied","Data":"bc41147600cb4c4e4138c061184ec4368edf6e5c7549ab1f77b5257dda77709f"}
Oct 31 16:42:32.047836 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.047770    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc41147600cb4c4e4138c061184ec4368edf6e5c7549ab1f77b5257dda77709f"
Oct 31 16:42:32.049189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.049169    4519 generic.go:356] "Generic (PLEG): container finished" podID="4a555145-7546-430a-90c6-a4a96b4a5106" containerID="9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d" exitCode=0
Oct 31 16:42:32.049284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.049200    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" event={"ID":"4a555145-7546-430a-90c6-a4a96b4a5106","Type":"ContainerDied","Data":"9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d"}
Oct 31 16:42:32.049284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.049220    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"
Oct 31 16:42:32.049284 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.049236    4519 scope.go:122] "RemoveContainer" containerID="9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d"
Oct 31 16:42:32.049535 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.049227    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n" event={"ID":"4a555145-7546-430a-90c6-a4a96b4a5106","Type":"ContainerDied","Data":"21cfc8ecd2fdf2dc95b0f8285d6875f569a6d5b329dc9ad2356efbfb49c02d9b"}
Oct 31 16:42:32.059886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.059866    4519 scope.go:122] "RemoveContainer" containerID="9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d"
Oct 31 16:42:32.060226 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:32.060200    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d\": container with ID starting with 9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d not found: ID does not exist" containerID="9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d"
Oct 31 16:42:32.060306 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.060236    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d"} err="failed to get container status \"9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d\": rpc error: code = NotFound desc = could not find container \"9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d\": container with ID starting with 9d586c1482340c543bad1521f14254f58b177f3097b556557fdad00c39be502d not found: ID does not exist"
Oct 31 16:42:32.120190 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.120162    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hwgc\" (UniqueName: \"kubernetes.io/projected/4a555145-7546-430a-90c6-a4a96b4a5106-kube-api-access-9hwgc\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:32.204737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.204710    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"]
Oct 31 16:42:32.210508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.210477    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ef6d93-847d6594c9-xdh7n"]
Oct 31 16:42:32.439663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:32.439590    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" path="/var/lib/kubelet/pods/4a555145-7546-430a-90c6-a4a96b4a5106/volumes"
Oct 31 16:42:34.462906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.462871    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main" containerID="cri-o://9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400" gracePeriod=45
Oct 31 16:42:34.760375 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.760348    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:42:34.849077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849045    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir\") pod \"15bdcd3c-9b39-4953-a2d4-651157576b56\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") "
Oct 31 16:42:34.849275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849092    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm\") pod \"15bdcd3c-9b39-4953-a2d4-651157576b56\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") "
Oct 31 16:42:34.849275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849139    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs\" (UniqueName: \"kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs\") pod \"15bdcd3c-9b39-4953-a2d4-651157576b56\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") "
Oct 31 16:42:34.849275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849173    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home\") pod \"15bdcd3c-9b39-4953-a2d4-651157576b56\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") "
Oct 31 16:42:34.849275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849262    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn\" (UniqueName: \"kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn\") pod \"15bdcd3c-9b39-4953-a2d4-651157576b56\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") "
Oct 31 16:42:34.849538 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849351    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir" pod "15bdcd3c-9b39-4953-a2d4-651157576b56" (UID: "15bdcd3c-9b39-4953-a2d4-651157576b56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:34.849593 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849554    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home" pod "15bdcd3c-9b39-4953-a2d4-651157576b56" (UID: "15bdcd3c-9b39-4953-a2d4-651157576b56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:34.849684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849665    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:34.849684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.849682    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:34.851450 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.851426    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm" pod "15bdcd3c-9b39-4953-a2d4-651157576b56" (UID: "15bdcd3c-9b39-4953-a2d4-651157576b56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:34.851586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.851527    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs" pod "15bdcd3c-9b39-4953-a2d4-651157576b56" (UID: "15bdcd3c-9b39-4953-a2d4-651157576b56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:34.851586 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.851527    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn" pod "15bdcd3c-9b39-4953-a2d4-651157576b56" (UID: "15bdcd3c-9b39-4953-a2d4-651157576b56"). InnerVolumeSpecName "kube-api-access-ktdnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:34.950154 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.950121    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache\") pod \"15bdcd3c-9b39-4953-a2d4-651157576b56\" (UID: \"15bdcd3c-9b39-4953-a2d4-651157576b56\") "
Oct 31 16:42:34.950432 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.950410    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache" pod "15bdcd3c-9b39-4953-a2d4-651157576b56" (UID: "15bdcd3c-9b39-4953-a2d4-651157576b56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:34.950557 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.950542    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktdnn\" (UniqueName: \"kubernetes.io/projected/15bdcd3c-9b39-4953-a2d4-651157576b56-kube-api-access-ktdnn\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:34.950594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.950560    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:34.950594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.950570    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15bdcd3c-9b39-4953-a2d4-651157576b56-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:34.950594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:34.950579    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15bdcd3c-9b39-4953-a2d4-651157576b56-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:35.074788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.074702    4519 generic.go:356] "Generic (PLEG): container finished" podID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerID="9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400" exitCode=0
Oct 31 16:42:35.074788 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.074758    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"
Oct 31 16:42:35.074978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.074794    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" event={"ID":"15bdcd3c-9b39-4953-a2d4-651157576b56","Type":"ContainerDied","Data":"9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400"}
Oct 31 16:42:35.074978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.074850    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf" event={"ID":"15bdcd3c-9b39-4953-a2d4-651157576b56","Type":"ContainerDied","Data":"9d8821b5b214dbb1c617f09f9980e0dadca366e0b08cad51406d3aeca47b3bdc"}
Oct 31 16:42:35.074978 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.074874    4519 scope.go:122] "RemoveContainer" containerID="9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400"
Oct 31 16:42:35.086934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.086903    4519 scope.go:122] "RemoveContainer" containerID="9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400"
Oct 31 16:42:35.087249 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:35.087227    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400\": container with ID starting with 9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400 not found: ID does not exist" containerID="9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400"
Oct 31 16:42:35.087334 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.087258    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400"} err="failed to get container status \"9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400\": rpc error: code = NotFound desc = could not find container \"9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400\": container with ID starting with 9d769ba9863ae9edf021d19dfd35da0c1e9c9964b2582371cb87b3109d344400 not found: ID does not exist"
Oct 31 16:42:35.193159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.193116    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"]
Oct 31 16:42:35.197758 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.197726    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ef6d93-2g5kn"]
Oct 31 16:42:35.238547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.238507    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"]
Oct 31 16:42:35.245613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:35.245585    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/premium-w6-a1685e-kserve-8688dcc666-c4nvf"]
Oct 31 16:42:36.444823 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:36.444790    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" path="/var/lib/kubelet/pods/15bdcd3c-9b39-4953-a2d4-651157576b56/volumes"
Oct 31 16:42:36.445275 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:36.445260    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a534710-564e-4582-98cc-40927fb90207" path="/var/lib/kubelet/pods/1a534710-564e-4582-98cc-40927fb90207/volumes"
Oct 31 16:42:36.524181 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:36.524147    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"]
Oct 31 16:42:36.577297 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:36.574173    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ac1273-jdp2l"]
Oct 31 16:42:38.437863 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:38.437830    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9817cb-7b05-4517-93a4-daf63de06c45" path="/var/lib/kubelet/pods/dd9817cb-7b05-4517-93a4-daf63de06c45/volumes"
Oct 31 16:42:38.892067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:38.892040    4519 scope.go:122] "RemoveContainer" containerID="349bf0d11d73d62cc5f7e45cd6926cdd826d641977aa9aaa5377ba0f129994ec"
Oct 31 16:42:38.901406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:38.901383    4519 scope.go:122] "RemoveContainer" containerID="468c02d75ce77541ca7b250f5555f69023bd98d5f1a93052b884e0925fe6f818"
Oct 31 16:42:38.910793 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:38.910773    4519 scope.go:122] "RemoveContainer" containerID="cd5ab88086d1b9bd4d4e7592b1a224ec164e645594d0774e7eb2ef86d4d1cf3e"
Oct 31 16:42:43.019691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:43.019655    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"]
Oct 31 16:42:43.020229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:43.019858    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy" containerID="cri-o://b139a62818aa0df6503c1be5dce56bcb2a0607ce00776c5b70d42a90ad631802" gracePeriod=30
Oct 31 16:42:48.166080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.166034    4519 generic.go:356] "Generic (PLEG): container finished" podID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerID="b139a62818aa0df6503c1be5dce56bcb2a0607ce00776c5b70d42a90ad631802" exitCode=0
Oct 31 16:42:48.166479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.166107    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" event={"ID":"ef1d4f62-fb25-460a-a12c-02ff0065a5f7","Type":"ContainerDied","Data":"b139a62818aa0df6503c1be5dce56bcb2a0607ce00776c5b70d42a90ad631802"}
Oct 31 16:42:48.274719 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.274693    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:42:48.388118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388034    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388104    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388137    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388217    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388267    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388297    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388346    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388370    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:48.388720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388409    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388505    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert\") pod \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\" (UID: \"ef1d4f62-fb25-460a-a12c-02ff0065a5f7\") "
Oct 31 16:42:48.388720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388547    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:48.388720 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388614    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:48.388933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388824    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.388933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388837    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.388933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.388847    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.389106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.389084    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:48.389215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.389189    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:48.390528 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.390505    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:42:48.390652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.390634    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:48.390779 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.390757    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:48.390872 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.390855    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc" pod "ef1d4f62-fb25-460a-a12c-02ff0065a5f7" (UID: "ef1d4f62-fb25-460a-a12c-02ff0065a5f7"). InnerVolumeSpecName "kube-api-access-pc6nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:48.489641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.489608    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6nc\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-kube-api-access-pc6nc\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.489641 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.489644    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.489853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.489660    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.489853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.489674    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.489853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.489688    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:48.489853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:48.489704    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1d4f62-fb25-460a-a12c-02ff0065a5f7-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:49.173541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:49.173511    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"
Oct 31 16:42:49.173541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:49.173528    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt" event={"ID":"ef1d4f62-fb25-460a-a12c-02ff0065a5f7","Type":"ContainerDied","Data":"875e06f36d70868d4ba2c4694ac78dc4c9041d5013bf03e06d9e4d3dd6ec49a7"}
Oct 31 16:42:49.174014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:49.173599    4519 scope.go:122] "RemoveContainer" containerID="b139a62818aa0df6503c1be5dce56bcb2a0607ce00776c5b70d42a90ad631802"
Oct 31 16:42:49.333159 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:49.333110    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"]
Oct 31 16:42:49.346401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:49.346369    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-5d4lt"]
Oct 31 16:42:49.425914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:49.425835    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:42:50.443291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:42:50.443254    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" path="/var/lib/kubelet/pods/ef1d4f62-fb25-460a-a12c-02ff0065a5f7/volumes"
Oct 31 16:42:54.003552 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:42:54.003517    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:43:13.377484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377439    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"]
Oct 31 16:43:13.377918 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377902    4519 cpu_manager.go:414] "containerMap: removing container" podUID="449026bc-7a74-4640-b65f-564a85c04705" containerName="revoke-keys"
Oct 31 16:43:13.377965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377920    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449026bc-7a74-4640-b65f-564a85c04705" containerName="revoke-keys"
Oct 31 16:43:13.377965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377942    4519 cpu_manager.go:414] "containerMap: removing container" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main"
Oct 31 16:43:13.377965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377947    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main"
Oct 31 16:43:13.377965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377961    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377967    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377974    4519 cpu_manager.go:414] "containerMap: removing container" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377979    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.377998    4519 cpu_manager.go:414] "containerMap: removing container" podUID="5596b388-86a8-4ba7-86d2-11f50a01d63f" containerName="revoke-keys"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.378003    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5596b388-86a8-4ba7-86d2-11f50a01d63f" containerName="revoke-keys"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.378063    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="5596b388-86a8-4ba7-86d2-11f50a01d63f" containerName="revoke-keys"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.378086    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="449026bc-7a74-4640-b65f-564a85c04705" containerName="revoke-keys"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.378093    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bdcd3c-9b39-4953-a2d4-651157576b56" containerName="main"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.378101    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a555145-7546-430a-90c6-a4a96b4a5106" containerName="payload-processing"
Oct 31 16:43:13.378112 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.378107    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1d4f62-fb25-460a-a12c-02ff0065a5f7" containerName="istio-proxy"
Oct 31 16:43:13.381612 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.381594    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.432438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.432417    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a1685e-openshift-default-dockercfg-2cr54\""
Oct 31 16:43:13.482682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.482652    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"]
Oct 31 16:43:13.565648 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565617    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.565864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565658    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.565864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565683    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.565864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565702    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.565864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565821    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.566058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565867    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.566058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.565963    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.566058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.566009    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.566058 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.566028    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spm9\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.666866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.666776    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.666866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.666832    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.666866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.666859    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.666877    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667178    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667316 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667234    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667334    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667378 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667357    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667443    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667509    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spm9\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.667654 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667641    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.668046 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.667865    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.668330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.668295    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.668587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.668563    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.669410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.669389    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.670207 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.670189    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.722572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.722535    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.722760 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.722695    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spm9\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:13.995131 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:13.995099    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:14.258752 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:14.258655    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"]
Oct 31 16:43:14.420166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:14.420140    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"]
Oct 31 16:43:14.420716 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:43:14.420689    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4941e22_762c_49ea_9a3f_83ce1d27034f.slice/crio-890b9b261362d5463247a84ed7c1c792c378acccf167aecd1917512b5ea26dcf WatchSource:0}: Error finding container 890b9b261362d5463247a84ed7c1c792c378acccf167aecd1917512b5ea26dcf: Status 404 returned error can't find the container with id 890b9b261362d5463247a84ed7c1c792c378acccf167aecd1917512b5ea26dcf
Oct 31 16:43:14.423199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:14.423167    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:43:14.423286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:14.423232    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:43:14.423286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:14.423260    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:43:15.400287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:15.400254    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w" event={"ID":"c4941e22-762c-49ea-9a3f-83ce1d27034f","Type":"ContainerStarted","Data":"87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215"}
Oct 31 16:43:15.400530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:15.400310    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w" event={"ID":"c4941e22-762c-49ea-9a3f-83ce1d27034f","Type":"ContainerStarted","Data":"890b9b261362d5463247a84ed7c1c792c378acccf167aecd1917512b5ea26dcf"}
Oct 31 16:43:15.400530 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:15.400317    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w" podUID="c4941e22-762c-49ea-9a3f-83ce1d27034f" containerName="istio-proxy" containerID="cri-o://87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215" gracePeriod=30
Oct 31 16:43:15.607274 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:15.607223    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w" podStartSLOduration=2.607209738 podStartE2EDuration="2.607209738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:43:15.557593789 +0000 UTC m=+2439.257523119" watchObservedRunningTime="2026-10-31 16:43:15.607209738 +0000 UTC m=+2439.307139043"
Oct 31 16:43:20.653650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.653620    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:20.741231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741198    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741264    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741317    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741352    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741392    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741435 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741419    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741482    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741513    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741543    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data\") pod \"c4941e22-762c-49ea-9a3f-83ce1d27034f\" (UID: \"c4941e22-762c-49ea-9a3f-83ce1d27034f\") "
Oct 31 16:43:20.741767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741629    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:20.741767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741740    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:20.742050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741818    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:43:20.742050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741857    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:20.742050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.741972    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:20.742050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.742017    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.742050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.742036    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4941e22-762c-49ea-9a3f-83ce1d27034f-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.742050 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.742052    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.742401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.742064    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.743604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.743577    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:43:20.743756 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.743739    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:20.743988 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.743964    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:43:20.743988 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.743969    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9" pod "c4941e22-762c-49ea-9a3f-83ce1d27034f" (UID: "c4941e22-762c-49ea-9a3f-83ce1d27034f"). InnerVolumeSpecName "kube-api-access-7spm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:43:20.843331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.843290    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.843331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.843331    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.843331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.843344    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spm9\" (UniqueName: \"kubernetes.io/projected/c4941e22-762c-49ea-9a3f-83ce1d27034f-kube-api-access-7spm9\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.843613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.843354    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.843613 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:20.843365    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4941e22-762c-49ea-9a3f-83ce1d27034f-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:21.440155 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.440120    4519 generic.go:356] "Generic (PLEG): container finished" podID="c4941e22-762c-49ea-9a3f-83ce1d27034f" containerID="87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215" exitCode=0
Oct 31 16:43:21.440332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.440164    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"
Oct 31 16:43:21.440332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.440201    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w" event={"ID":"c4941e22-762c-49ea-9a3f-83ce1d27034f","Type":"ContainerDied","Data":"87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215"}
Oct 31 16:43:21.440332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.440252    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w" event={"ID":"c4941e22-762c-49ea-9a3f-83ce1d27034f","Type":"ContainerDied","Data":"890b9b261362d5463247a84ed7c1c792c378acccf167aecd1917512b5ea26dcf"}
Oct 31 16:43:21.440332 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.440278    4519 scope.go:122] "RemoveContainer" containerID="87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215"
Oct 31 16:43:21.450604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.450583    4519 scope.go:122] "RemoveContainer" containerID="87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215"
Oct 31 16:43:21.450882 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:43:21.450864    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215\": container with ID starting with 87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215 not found: ID does not exist" containerID="87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215"
Oct 31 16:43:21.450927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.450891    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215"} err="failed to get container status \"87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215\": rpc error: code = NotFound desc = could not find container \"87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215\": container with ID starting with 87232350d579c986074f62a22d20b76524fabad6d6b1135c050d74524c000215 not found: ID does not exist"
Oct 31 16:43:21.643118 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.643079    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"]
Oct 31 16:43:21.690219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:21.690135    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-v649w"]
Oct 31 16:43:22.438016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:22.437980    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4941e22-762c-49ea-9a3f-83ce1d27034f" path="/var/lib/kubelet/pods/c4941e22-762c-49ea-9a3f-83ce1d27034f/volumes"
Oct 31 16:43:24.004159 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:43:24.004115    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:43:24.085044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:24.085008    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"]
Oct 31 16:43:24.087359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:24.087335    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cfef75-shzcb"]
Oct 31 16:43:24.437938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:24.437851    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377fbaab-69ff-4f9b-980a-92a9a0654976" path="/var/lib/kubelet/pods/377fbaab-69ff-4f9b-980a-92a9a0654976/volumes"
Oct 31 16:43:31.183711 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:31.183670    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:43:38.982403 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:38.982365    4519 scope.go:122] "RemoveContainer" containerID="82144edd2245b95f218db777fbf763d49cba662753a6cb28b7d3dc7649827624"
Oct 31 16:43:38.992973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:38.992949    4519 scope.go:122] "RemoveContainer" containerID="c108a4f1516b6039ed2e3fac3d192038bef9a8f9eb30f506f3aff72107ad3650"
Oct 31 16:43:39.002326 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:39.002296    4519 scope.go:122] "RemoveContainer" containerID="aeda2b7312afcf1c9705d9455a1188ee85ba0f3c3b5fc3503e3add2a3fc58fe1"
Oct 31 16:43:43.695513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:43:43.695477    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:43:54.003720 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:43:54.003683    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:44:24.003472 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:44:24.003421    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:44:35.565302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:35.565263    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:44:39.041956 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:39.041922    4519 scope.go:122] "RemoveContainer" containerID="bca701bbf4a396fa48280a84e5b781fe9f10c9e4a6c2d7b2122cab0814f4a382"
Oct 31 16:44:39.051811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:39.051786    4519 scope.go:122] "RemoveContainer" containerID="b1fe9841afd00693fedc454e0bda3b1c6db7062f480599ef723b9838e8ad2c3c"
Oct 31 16:44:39.062571 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:39.062548    4519 scope.go:122] "RemoveContainer" containerID="886177cbfe060752d887d7ecf70f37ce6d1b0456cd86e2996889bbf155e12d8a"
Oct 31 16:44:43.044302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:43.044264    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"]
Oct 31 16:44:43.091493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:43.091463    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c009319b-fnwj6"]
Oct 31 16:44:43.093749 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:43.093737    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"]
Oct 31 16:44:43.095726 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:43.095703    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d97adbbd-qnl4q"]
Oct 31 16:44:44.437507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:44.437448    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec88070-714e-419e-a0ad-0799a40c9cac" path="/var/lib/kubelet/pods/1ec88070-714e-419e-a0ad-0799a40c9cac/volumes"
Oct 31 16:44:44.437944 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:44:44.437929    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3197fc09-35f8-44fe-80b1-e818f2479d06" path="/var/lib/kubelet/pods/3197fc09-35f8-44fe-80b1-e818f2479d06/volumes"
Oct 31 16:44:54.024491 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:44:54.024425    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:45:00.163246 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.163208    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"]
Oct 31 16:45:00.163954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.163930    4519 cpu_manager.go:414] "containerMap: removing container" podUID="c4941e22-762c-49ea-9a3f-83ce1d27034f" containerName="istio-proxy"
Oct 31 16:45:00.164065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.163956    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4941e22-762c-49ea-9a3f-83ce1d27034f" containerName="istio-proxy"
Oct 31 16:45:00.164141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.164062    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4941e22-762c-49ea-9a3f-83ce1d27034f" containerName="istio-proxy"
Oct 31 16:45:00.169352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.169328    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"]
Oct 31 16:45:00.169493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.169463    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.173145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.173126    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.218772 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.218742    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:45:00.263067 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.263029    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx47d\" (UniqueName: \"kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d\") pod \"maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.263260 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.263083    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891085-9stpm\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.263320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.263249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.263320 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.263309    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthvd\" (UniqueName: \"kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd\") pod \"maas-api-key-cleanup-29891085-9stpm\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.266104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.266084    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:45:00.266229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.266204    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:45:00.269166 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.269142    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"]
Oct 31 16:45:00.272933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.272912    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"]
Oct 31 16:45:00.364255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.364221    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx47d\" (UniqueName: \"kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d\") pod \"maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.364512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.364267    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891085-9stpm\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.364512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.364320    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.364512 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.364359    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthvd\" (UniqueName: \"kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd\") pod \"maas-api-key-cleanup-29891085-9stpm\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.364937 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.364918    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891085-9stpm\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.365014 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.364996    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.418285 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.418211    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthvd\" (UniqueName: \"kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd\") pod \"maas-api-key-cleanup-29891085-9stpm\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.418421 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.418372    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx47d\" (UniqueName: \"kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d\") pod \"maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.483904 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.483876    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:00.489710 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.489684    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:00.854962 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.854931    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"]
Oct 31 16:45:00.861397 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:45:00.861372    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b71ba4c_ccfb_46d8_ba3b_cd9f67e348b3.slice/crio-af4e1ae825279edfceca3ee04709c7cd7c35de6c9feb6cf6ccb92eac1f592935 WatchSource:0}: Error finding container af4e1ae825279edfceca3ee04709c7cd7c35de6c9feb6cf6ccb92eac1f592935: Status 404 returned error can't find the container with id af4e1ae825279edfceca3ee04709c7cd7c35de6c9feb6cf6ccb92eac1f592935
Oct 31 16:45:00.906915 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:00.906884    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"]
Oct 31 16:45:00.912417 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:45:00.912390    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2dc8b6_888b_4a86_8b8c_6fbdb22cf096.slice/crio-b0e271c764fdc0d219f9a0881d600c21c43ff621f1fe75c4f56cbc03e222992f WatchSource:0}: Error finding container b0e271c764fdc0d219f9a0881d600c21c43ff621f1fe75c4f56cbc03e222992f: Status 404 returned error can't find the container with id b0e271c764fdc0d219f9a0881d600c21c43ff621f1fe75c4f56cbc03e222992f
Oct 31 16:45:01.040740 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:01.040701    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" event={"ID":"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3","Type":"ContainerStarted","Data":"af4e1ae825279edfceca3ee04709c7cd7c35de6c9feb6cf6ccb92eac1f592935"}
Oct 31 16:45:01.041905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:01.041880    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" event={"ID":"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096","Type":"ContainerStarted","Data":"b0e271c764fdc0d219f9a0881d600c21c43ff621f1fe75c4f56cbc03e222992f"}
Oct 31 16:45:02.049646 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:02.049610    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerID="57a3ca36bc5d4bd774825d9e78a1dad346e56febbfcd8e41e627aaa5502b59a5" exitCode=60
Oct 31 16:45:02.050123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:02.049693    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" event={"ID":"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096","Type":"ContainerDied","Data":"57a3ca36bc5d4bd774825d9e78a1dad346e56febbfcd8e41e627aaa5502b59a5"}
Oct 31 16:45:02.050123 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:02.049962    4519 scope.go:122] "RemoveContainer" containerID="57a3ca36bc5d4bd774825d9e78a1dad346e56febbfcd8e41e627aaa5502b59a5"
Oct 31 16:45:02.051427 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:02.051401    4519 generic.go:356] "Generic (PLEG): container finished" podID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerID="8b96015c411ba6e99bd2b4d9348c958d299a3c013922213d6679a96edec05eb2" exitCode=60
Oct 31 16:45:02.051550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:02.051437    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" event={"ID":"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3","Type":"ContainerDied","Data":"8b96015c411ba6e99bd2b4d9348c958d299a3c013922213d6679a96edec05eb2"}
Oct 31 16:45:02.051680 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:02.051667    4519 scope.go:122] "RemoveContainer" containerID="8b96015c411ba6e99bd2b4d9348c958d299a3c013922213d6679a96edec05eb2"
Oct 31 16:45:03.059467 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.059424    4519 generic.go:356] "Generic (PLEG): container finished" podID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerID="9250bf78c294095d2e99939fbe5670279332396275de356d7a30f0c70941f74e" exitCode=60
Oct 31 16:45:03.059923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.059517    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" event={"ID":"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3","Type":"ContainerDied","Data":"9250bf78c294095d2e99939fbe5670279332396275de356d7a30f0c70941f74e"}
Oct 31 16:45:03.059923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.059580    4519 scope.go:122] "RemoveContainer" containerID="8b96015c411ba6e99bd2b4d9348c958d299a3c013922213d6679a96edec05eb2"
Oct 31 16:45:03.059923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.059861    4519 scope.go:122] "RemoveContainer" containerID="9250bf78c294095d2e99939fbe5670279332396275de356d7a30f0c70941f74e"
Oct 31 16:45:03.060165 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:03.060141    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm_odh-ai-gateway-infra(0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3"
Oct 31 16:45:03.061604 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.061585    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerID="19c9e418437fc1e90a11fde387332780f11a48ffcef13e68fd3f26fc5f481aaf" exitCode=60
Oct 31 16:45:03.061696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.061666    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" event={"ID":"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096","Type":"ContainerDied","Data":"19c9e418437fc1e90a11fde387332780f11a48ffcef13e68fd3f26fc5f481aaf"}
Oct 31 16:45:03.061853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.061837    4519 scope.go:122] "RemoveContainer" containerID="19c9e418437fc1e90a11fde387332780f11a48ffcef13e68fd3f26fc5f481aaf"
Oct 31 16:45:03.062065 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:03.062042    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891085-9stpm_odh-ai-gateway-infra(4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096"
Oct 31 16:45:03.071313 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:03.071294    4519 scope.go:122] "RemoveContainer" containerID="57a3ca36bc5d4bd774825d9e78a1dad346e56febbfcd8e41e627aaa5502b59a5"
Oct 31 16:45:04.069923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:04.069892    4519 scope.go:122] "RemoveContainer" containerID="19c9e418437fc1e90a11fde387332780f11a48ffcef13e68fd3f26fc5f481aaf"
Oct 31 16:45:04.070426 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:04.070192    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891085-9stpm_odh-ai-gateway-infra(4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096"
Oct 31 16:45:04.071663 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:04.071640    4519 scope.go:122] "RemoveContainer" containerID="9250bf78c294095d2e99939fbe5670279332396275de356d7a30f0c70941f74e"
Oct 31 16:45:04.071909 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:04.071886    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm_odh-ai-gateway-infra(0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3"
Oct 31 16:45:05.043279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:05.043243    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"]
Oct 31 16:45:05.090225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:05.090192    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6896399a-mkl2z"]
Oct 31 16:45:06.437702 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:06.437670    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cce11a5-b03b-4066-bda6-47a95ec657c4" path="/var/lib/kubelet/pods/4cce11a5-b03b-4066-bda6-47a95ec657c4/volumes"
Oct 31 16:45:13.431715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:13.431686    4519 scope.go:122] "RemoveContainer" containerID="9250bf78c294095d2e99939fbe5670279332396275de356d7a30f0c70941f74e"
Oct 31 16:45:14.133965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:14.133917    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" event={"ID":"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3","Type":"ContainerStarted","Data":"9f5e716a2d923d24209e929c2aa2b455132febca5715b995feb1b3280ddc6577"}
Oct 31 16:45:14.245743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:14.245695    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" podStartSLOduration=14.245680829 podStartE2EDuration="14.245680829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:45:14.243488376 +0000 UTC m=+2557.943417680" watchObservedRunningTime="2026-10-31 16:45:14.245680829 +0000 UTC m=+2557.945610139"
Oct 31 16:45:15.140748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:15.140713    4519 generic.go:356] "Generic (PLEG): container finished" podID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerID="9f5e716a2d923d24209e929c2aa2b455132febca5715b995feb1b3280ddc6577" exitCode=60
Oct 31 16:45:15.141188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:15.140787    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" event={"ID":"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3","Type":"ContainerDied","Data":"9f5e716a2d923d24209e929c2aa2b455132febca5715b995feb1b3280ddc6577"}
Oct 31 16:45:15.141188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:15.140852    4519 scope.go:122] "RemoveContainer" containerID="9250bf78c294095d2e99939fbe5670279332396275de356d7a30f0c70941f74e"
Oct 31 16:45:15.141188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:15.141132    4519 scope.go:122] "RemoveContainer" containerID="9f5e716a2d923d24209e929c2aa2b455132febca5715b995feb1b3280ddc6577"
Oct 31 16:45:15.141404 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:15.141379    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm_odh-ai-gateway-infra(0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3"
Oct 31 16:45:15.262090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:15.262059    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"]
Oct 31 16:45:15.431707 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:15.431632    4519 scope.go:122] "RemoveContainer" containerID="19c9e418437fc1e90a11fde387332780f11a48ffcef13e68fd3f26fc5f481aaf"
Oct 31 16:45:16.148009 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.147973    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" event={"ID":"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096","Type":"ContainerStarted","Data":"d384845f64e7df9f9c6eb0e0f99dcc68f12dca8596293372c62de2b37110a7af"}
Oct 31 16:45:16.256668 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.256614    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" podStartSLOduration=16.256599986 podStartE2EDuration="16.256599986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:45:16.255251167 +0000 UTC m=+2559.955180480" watchObservedRunningTime="2026-10-31 16:45:16.256599986 +0000 UTC m=+2559.956529297"
Oct 31 16:45:16.295564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.295538    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:16.425026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.424956    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca\") pod \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") "
Oct 31 16:45:16.425026 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.425020    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d\" (UniqueName: \"kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d\") pod \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\" (UID: \"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3\") "
Oct 31 16:45:16.425299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.425275    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca" pod "0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" (UID: "0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:45:16.425372 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.425361    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:45:16.426954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.426926    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d" pod "0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" (UID: "0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3"). InnerVolumeSpecName "kube-api-access-hx47d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:45:16.526673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:16.526639    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx47d\" (UniqueName: \"kubernetes.io/projected/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3-kube-api-access-hx47d\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:45:17.158382 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.158339    4519 generic.go:356] "Generic (PLEG): container finished" podID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerID="d384845f64e7df9f9c6eb0e0f99dcc68f12dca8596293372c62de2b37110a7af" exitCode=60
Oct 31 16:45:17.158840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.158400    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" event={"ID":"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096","Type":"ContainerDied","Data":"d384845f64e7df9f9c6eb0e0f99dcc68f12dca8596293372c62de2b37110a7af"}
Oct 31 16:45:17.158840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.158479    4519 scope.go:122] "RemoveContainer" containerID="19c9e418437fc1e90a11fde387332780f11a48ffcef13e68fd3f26fc5f481aaf"
Oct 31 16:45:17.158840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.158801    4519 scope.go:122] "RemoveContainer" containerID="d384845f64e7df9f9c6eb0e0f99dcc68f12dca8596293372c62de2b37110a7af"
Oct 31 16:45:17.159129 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:17.159102    4519 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891085-9stpm_odh-ai-gateway-infra(4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096"
Oct 31 16:45:17.160044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.160015    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm" event={"ID":"0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3","Type":"ContainerDied","Data":"af4e1ae825279edfceca3ee04709c7cd7c35de6c9feb6cf6ccb92eac1f592935"}
Oct 31 16:45:17.160044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.160042    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"
Oct 31 16:45:17.170311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.170279    4519 scope.go:122] "RemoveContainer" containerID="9f5e716a2d923d24209e929c2aa2b455132febca5715b995feb1b3280ddc6577"
Oct 31 16:45:17.265541 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.265508    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"]
Oct 31 16:45:17.384477 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.384386    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"]
Oct 31 16:45:17.391310 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:17.391284    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-05db63-29891085-t9qwm"]
Oct 31 16:45:18.321053 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.321023    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:18.343038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.343013    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd\" (UniqueName: \"kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd\") pod \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") "
Oct 31 16:45:18.343195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.343127    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca\") pod \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\" (UID: \"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096\") "
Oct 31 16:45:18.343467 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.343433    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca" pod "4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" (UID: "4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:45:18.345025 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.345005    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd" pod "4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" (UID: "4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096"). InnerVolumeSpecName "kube-api-access-sthvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:45:18.440222 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.440142    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" path="/var/lib/kubelet/pods/0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3/volumes"
Oct 31 16:45:18.444490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.444448    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthvd\" (UniqueName: \"kubernetes.io/projected/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-kube-api-access-sthvd\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:45:18.444490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:18.444489    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096-maas-api-service-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:45:19.175478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:19.175425    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"
Oct 31 16:45:19.175693 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:19.175424    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm" event={"ID":"4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096","Type":"ContainerDied","Data":"b0e271c764fdc0d219f9a0881d600c21c43ff621f1fe75c4f56cbc03e222992f"}
Oct 31 16:45:19.175693 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:19.175571    4519 scope.go:122] "RemoveContainer" containerID="d384845f64e7df9f9c6eb0e0f99dcc68f12dca8596293372c62de2b37110a7af"
Oct 31 16:45:19.330999 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:19.330964    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"]
Oct 31 16:45:19.333287 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:19.333266    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891085-9stpm"]
Oct 31 16:45:20.438661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:20.438625    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" path="/var/lib/kubelet/pods/4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096/volumes"
Oct 31 16:45:24.002808 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:24.002762    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:45:33.044660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:33.044626    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"]
Oct 31 16:45:33.091796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:33.091759    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"]
Oct 31 16:45:33.097736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:33.097698    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fce78cad-zgcg9"]
Oct 31 16:45:33.100690 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:33.100662    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786e30b9-89scl"]
Oct 31 16:45:34.437369 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:34.437334    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449de57b-fa3e-4296-b562-9002efea6e4e" path="/var/lib/kubelet/pods/449de57b-fa3e-4296-b562-9002efea6e4e/volumes"
Oct 31 16:45:34.437837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:34.437818    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6f27f6-5ed8-4493-97d2-72d86457e796" path="/var/lib/kubelet/pods/ca6f27f6-5ed8-4493-97d2-72d86457e796/volumes"
Oct 31 16:45:39.092339 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:39.092300    4519 scope.go:122] "RemoveContainer" containerID="43fc24387ee02fec6bf97f41e7d6cf594f60d1cbf0bcbba5e90bbac27c370fd2"
Oct 31 16:45:39.101312 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:39.101293    4519 scope.go:122] "RemoveContainer" containerID="861a86794e6860d8d320ed92fbe83c2af663f62e601c8f3439ec24c7e0eb9aee"
Oct 31 16:45:39.109219 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:39.109200    4519 scope.go:122] "RemoveContainer" containerID="598d08f18b3fb67243ef34cb6b114555885befbc5d1cb1c586e10c5be3d0546c"
Oct 31 16:45:39.116899 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:39.116880    4519 scope.go:122] "RemoveContainer" containerID="b0137266447d16317ed093f4e80e782dd9a7c7dc8b4d019ad6c9b504413f1d6e"
Oct 31 16:45:45.046990 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:45.046958    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"]
Oct 31 16:45:45.093873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:45.093842    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"]
Oct 31 16:45:45.095826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:45.095814    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5f8567-qhrzz"]
Oct 31 16:45:45.097652 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:45.097629    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b84ba9f-m8gzp"]
Oct 31 16:45:46.437143 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:46.437111    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208030bb-9d6a-4b0a-91f6-e81fd8e615b5" path="/var/lib/kubelet/pods/208030bb-9d6a-4b0a-91f6-e81fd8e615b5/volumes"
Oct 31 16:45:46.437550 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:46.437534    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f465efc6-db89-4a9e-b02e-5cc199f72782" path="/var/lib/kubelet/pods/f465efc6-db89-4a9e-b02e-5cc199f72782/volumes"
Oct 31 16:45:54.003403 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:45:54.003368    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:45:56.003685 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.003656    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"]
Oct 31 16:45:56.004088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004076    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004089    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004101    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004108    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004118    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004123    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004135 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004133    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004138    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004145    4519 cpu_manager.go:414] "containerMap: removing container" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004150    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004161    4519 cpu_manager.go:414] "containerMap: removing container" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004166    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004260    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004268    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004275    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004281    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004285    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b71ba4c-ccfb-46d8-ba3b-cd9f67e348b3" containerName="cleanup"
Oct 31 16:45:56.004331 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.004291    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2dc8b6-888b-4a86-8b8c-6fbdb22cf096" containerName="cleanup"
Oct 31 16:45:56.016367 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.016335    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.059572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.059538    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"]
Oct 31 16:45:56.065855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.065827    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:45:56.104070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.104041    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:45:56.104263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.104238    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:45:56.113164 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.113145    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:45:56.133680 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.133657    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvp75\" (UniqueName: \"kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.133816 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.133702    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.133888 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.133840    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.234697 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.234657    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.234893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.234718    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.234893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.234791    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp75\" (UniqueName: \"kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.235390 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.235366    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.237010 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.236992    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.288607 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.288539    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp75\" (UniqueName: \"kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75\") pod \"maas-api-revoke-sub-86f89a9b-zb28v\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.329494 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.329448    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:56.701653 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:56.701571    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"]
Oct 31 16:45:56.707664 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:45:56.707637    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cfb6e6_8fb7_4f99_8cdf_441fc3ac5c12.slice/crio-99e4eac21e3125918941f1df5839d3936f4bd53018dccb4f7a5474935eefa9f6 WatchSource:0}: Error finding container 99e4eac21e3125918941f1df5839d3936f4bd53018dccb4f7a5474935eefa9f6: Status 404 returned error can't find the container with id 99e4eac21e3125918941f1df5839d3936f4bd53018dccb4f7a5474935eefa9f6
Oct 31 16:45:57.406552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:57.406447    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v" event={"ID":"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12","Type":"ContainerStarted","Data":"f066e33920b98886ec2756f96b5881b56caa47be4f16e165a645c5987d3a955d"}
Oct 31 16:45:57.406552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:57.406527    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v" event={"ID":"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12","Type":"ContainerStarted","Data":"99e4eac21e3125918941f1df5839d3936f4bd53018dccb4f7a5474935eefa9f6"}
Oct 31 16:45:58.413748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:58.413714    4519 generic.go:356] "Generic (PLEG): container finished" podID="a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" containerID="f066e33920b98886ec2756f96b5881b56caa47be4f16e165a645c5987d3a955d" exitCode=0
Oct 31 16:45:58.414169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:58.413802    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v" event={"ID":"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12","Type":"ContainerDied","Data":"f066e33920b98886ec2756f96b5881b56caa47be4f16e165a645c5987d3a955d"}
Oct 31 16:45:59.553611 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.553587    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:45:59.671829 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.671754    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75\" (UniqueName: \"kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75\") pod \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") "
Oct 31 16:45:59.672020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.671897    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca\") pod \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") "
Oct 31 16:45:59.672020 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.671991    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca\") pod \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\" (UID: \"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12\") "
Oct 31 16:45:59.672333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.672300    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca" pod "a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" (UID: "a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:45:59.674064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.674038    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75" pod "a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" (UID: "a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12"). InnerVolumeSpecName "kube-api-access-zvp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:45:59.674185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.674090    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca" pod "a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" (UID: "a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:45:59.773587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.773547    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvp75\" (UniqueName: \"kubernetes.io/projected/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-kube-api-access-zvp75\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:45:59.773587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.773579    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:45:59.773587 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:45:59.773593    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:00.427162 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:00.427128    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"
Oct 31 16:46:00.427356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:00.427164    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v" event={"ID":"a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12","Type":"ContainerDied","Data":"99e4eac21e3125918941f1df5839d3936f4bd53018dccb4f7a5474935eefa9f6"}
Oct 31 16:46:00.427356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:00.427245    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e4eac21e3125918941f1df5839d3936f4bd53018dccb4f7a5474935eefa9f6"
Oct 31 16:46:09.053066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.053030    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"]
Oct 31 16:46:09.053562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.053501    4519 cpu_manager.go:414] "containerMap: removing container" podUID="a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" containerName="revoke-keys"
Oct 31 16:46:09.053562 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.053514    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" containerName="revoke-keys"
Oct 31 16:46:09.053650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.053586    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" containerName="revoke-keys"
Oct 31 16:46:09.056932 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.056904    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.110302 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.110268    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"]
Oct 31 16:46:09.154361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.154333    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:46:09.154361 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.154332    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:46:09.154624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.154336    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:46:09.154624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.154376    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:46:09.175747 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.175715    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.175924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.175789    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmgpl\" (UniqueName: \"kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.175924 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.175909    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.276873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.276825    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgpl\" (UniqueName: \"kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.277065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.276914    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.277065 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.277011    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.277683 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.277656    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.279250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.279231    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.331893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.331828    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgpl\" (UniqueName: \"kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl\") pod \"maas-api-revoke-sub-1870493f-hdxxh\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.372523 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.372491    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:09.741657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:09.741619    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"]
Oct 31 16:46:09.746984 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:46:09.746949    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058aa6c4_b6e9_418b_a9cb_c9fc8b9e0e72.slice/crio-260521ae6b31ce0505ae138f0673207db5e9f9f25fbe6b81139134eb3bdf89af WatchSource:0}: Error finding container 260521ae6b31ce0505ae138f0673207db5e9f9f25fbe6b81139134eb3bdf89af: Status 404 returned error can't find the container with id 260521ae6b31ce0505ae138f0673207db5e9f9f25fbe6b81139134eb3bdf89af
Oct 31 16:46:10.486060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:10.486021    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh" event={"ID":"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72","Type":"ContainerStarted","Data":"89ba04642677a0878f054d918dbd59c254571c3d5160ddb77b1c8c9a65e65016"}
Oct 31 16:46:10.486547 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:10.486084    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh" event={"ID":"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72","Type":"ContainerStarted","Data":"260521ae6b31ce0505ae138f0673207db5e9f9f25fbe6b81139134eb3bdf89af"}
Oct 31 16:46:10.598449 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:10.598403    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh" podStartSLOduration=1.598389345 podStartE2EDuration="1.598389345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:46:10.595887722 +0000 UTC m=+2614.295817037" watchObservedRunningTime="2026-10-31 16:46:10.598389345 +0000 UTC m=+2614.298318656"
Oct 31 16:46:11.494019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:11.493984    4519 generic.go:356] "Generic (PLEG): container finished" podID="058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" containerID="89ba04642677a0878f054d918dbd59c254571c3d5160ddb77b1c8c9a65e65016" exitCode=0
Oct 31 16:46:11.494405 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:11.494072    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh" event={"ID":"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72","Type":"ContainerDied","Data":"89ba04642677a0878f054d918dbd59c254571c3d5160ddb77b1c8c9a65e65016"}
Oct 31 16:46:12.641059 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.641029    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:12.715437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.715403    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca\") pod \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") "
Oct 31 16:46:12.715643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.715446    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca\") pod \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") "
Oct 31 16:46:12.715643 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.715586    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl\" (UniqueName: \"kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl\") pod \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\" (UID: \"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72\") "
Oct 31 16:46:12.715875 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.715847    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca" pod "058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" (UID: "058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:46:12.717491 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.717467    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca" pod "058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" (UID: "058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:46:12.717624 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.717602    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl" pod "058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" (UID: "058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72"). InnerVolumeSpecName "kube-api-access-zmgpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:46:12.821279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.821185    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmgpl\" (UniqueName: \"kubernetes.io/projected/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-kube-api-access-zmgpl\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:12.824620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.824593    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:12.824820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.824805    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:12.843263 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:12.843228    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:46:13.506205 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:13.506175    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"
Oct 31 16:46:13.506401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:13.506179    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh" event={"ID":"058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72","Type":"ContainerDied","Data":"260521ae6b31ce0505ae138f0673207db5e9f9f25fbe6b81139134eb3bdf89af"}
Oct 31 16:46:13.506401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:13.506297    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="260521ae6b31ce0505ae138f0673207db5e9f9f25fbe6b81139134eb3bdf89af"
Oct 31 16:46:24.003282 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:46:24.003240    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:46:31.161330 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:31.161291    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:46:32.101364 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.101331    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"]
Oct 31 16:46:32.101796 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.101783    4519 cpu_manager.go:414] "containerMap: removing container" podUID="058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" containerName="revoke-keys"
Oct 31 16:46:32.101855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.101797    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" containerName="revoke-keys"
Oct 31 16:46:32.101903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.101894    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" containerName="revoke-keys"
Oct 31 16:46:32.105589 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.105569    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.156414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.156378    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"]
Oct 31 16:46:32.173820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.173787    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:46:32.173820 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.173803    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:46:32.174273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.173816    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:46:32.174273 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.174016    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:46:32.217237 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.217199    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45w7c\" (UniqueName: \"kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.217447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.217249    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.217447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.217340    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.318017 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.317980    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45w7c\" (UniqueName: \"kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.318289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.318032    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.318289 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.318054    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.318748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.318724    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.320354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.320333    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.372997 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.372918    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45w7c\" (UniqueName: \"kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c\") pod \"maas-api-revoke-keys-e2e-inf-a-05db63-kqqln\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.418560 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.418528    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:32.791051 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:32.791013    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"]
Oct 31 16:46:32.797498 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:46:32.797449    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063ced0c_f028_4527_a9d4_b398c2ac4a2e.slice/crio-e049bb4cb5cd11382e7cf808628a7fdc5f0a1ebd5f0c9abd2140d013037460d4 WatchSource:0}: Error finding container e049bb4cb5cd11382e7cf808628a7fdc5f0a1ebd5f0c9abd2140d013037460d4: Status 404 returned error can't find the container with id e049bb4cb5cd11382e7cf808628a7fdc5f0a1ebd5f0c9abd2140d013037460d4
Oct 31 16:46:33.624250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:33.624211    4519 generic.go:356] "Generic (PLEG): container finished" podID="063ced0c-f028-4527-a9d4-b398c2ac4a2e" containerID="7afb7345e965c308d154d37d236f4ce4509cb7e97f0dac509ce03a2ce0e7ae85" exitCode=0
Oct 31 16:46:33.624634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:33.624256    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln" event={"ID":"063ced0c-f028-4527-a9d4-b398c2ac4a2e","Type":"ContainerDied","Data":"7afb7345e965c308d154d37d236f4ce4509cb7e97f0dac509ce03a2ce0e7ae85"}
Oct 31 16:46:33.624634 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:33.624304    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln" event={"ID":"063ced0c-f028-4527-a9d4-b398c2ac4a2e","Type":"ContainerStarted","Data":"e049bb4cb5cd11382e7cf808628a7fdc5f0a1ebd5f0c9abd2140d013037460d4"}
Oct 31 16:46:34.770192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.770165    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:34.841673 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.841641    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c\" (UniqueName: \"kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c\") pod \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") "
Oct 31 16:46:34.841876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.841762    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca\") pod \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") "
Oct 31 16:46:34.841876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.841798    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca\") pod \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\" (UID: \"063ced0c-f028-4527-a9d4-b398c2ac4a2e\") "
Oct 31 16:46:34.842253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.842224    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca" pod "063ced0c-f028-4527-a9d4-b398c2ac4a2e" (UID: "063ced0c-f028-4527-a9d4-b398c2ac4a2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:46:34.843837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.843810    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c" pod "063ced0c-f028-4527-a9d4-b398c2ac4a2e" (UID: "063ced0c-f028-4527-a9d4-b398c2ac4a2e"). InnerVolumeSpecName "kube-api-access-45w7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:46:34.843925 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.843863    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca" pod "063ced0c-f028-4527-a9d4-b398c2ac4a2e" (UID: "063ced0c-f028-4527-a9d4-b398c2ac4a2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:46:34.942815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.942731    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45w7c\" (UniqueName: \"kubernetes.io/projected/063ced0c-f028-4527-a9d4-b398c2ac4a2e-kube-api-access-45w7c\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:34.942815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.942758    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:34.942815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:34.942767    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/063ced0c-f028-4527-a9d4-b398c2ac4a2e-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:35.637189 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:35.637148    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln" event={"ID":"063ced0c-f028-4527-a9d4-b398c2ac4a2e","Type":"ContainerDied","Data":"e049bb4cb5cd11382e7cf808628a7fdc5f0a1ebd5f0c9abd2140d013037460d4"}
Oct 31 16:46:35.637354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:35.637209    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e049bb4cb5cd11382e7cf808628a7fdc5f0a1ebd5f0c9abd2140d013037460d4"
Oct 31 16:46:35.637354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:35.637159    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"
Oct 31 16:46:39.161446 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.161413    4519 scope.go:122] "RemoveContainer" containerID="831b0f96644acfab7a7830ced5924475b980de44d03d41040e87955124e3ae28"
Oct 31 16:46:39.170658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.170638    4519 scope.go:122] "RemoveContainer" containerID="92910a8a95f44dea9133860c73c73df56319d3e4691e8f0516e655dacf316bd6"
Oct 31 16:46:39.179091 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.179065    4519 scope.go:122] "RemoveContainer" containerID="ccb5abfdb85e9411b25b29df10a2cde28844a2171e0c4fd58db48ea27fd6f7e2"
Oct 31 16:46:39.187008 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.186987    4519 scope.go:122] "RemoveContainer" containerID="92ecfc718ef2847050a5d2d5353f5b5f9925dcc7436a0bd882c3f542b0c3f979"
Oct 31 16:46:39.194828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.194809    4519 scope.go:122] "RemoveContainer" containerID="b892fd332a140817dcdbb8ff6290a951253bace450db182dc3a915d939fef478"
Oct 31 16:46:39.202365 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.202348    4519 scope.go:122] "RemoveContainer" containerID="94789620f5d23de87a6dd0921ced167e02cb2cdf863c1d53d0bb9651768e6c48"
Oct 31 16:46:39.209934 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:39.209917    4519 scope.go:122] "RemoveContainer" containerID="5447cbde274ddc82fc7e6cc2bf77af7d41c4865476b92e0058b71f52b7a2c4e7"
Oct 31 16:46:44.169853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:44.169816    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:46:54.004527 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:46:54.004486    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:46:59.322582 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:59.322547    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"]
Oct 31 16:46:59.369564 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:46:59.369529    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05db63-kqqln"]
Oct 31 16:47:00.438307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:00.438273    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063ced0c-f028-4527-a9d4-b398c2ac4a2e" path="/var/lib/kubelet/pods/063ced0c-f028-4527-a9d4-b398c2ac4a2e/volumes"
Oct 31 16:47:02.794815 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:02.794784    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:47:06.756257 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.756221    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"]
Oct 31 16:47:06.756753 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.756735    4519 cpu_manager.go:414] "containerMap: removing container" podUID="063ced0c-f028-4527-a9d4-b398c2ac4a2e" containerName="revoke-keys"
Oct 31 16:47:06.756811 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.756756    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063ced0c-f028-4527-a9d4-b398c2ac4a2e" containerName="revoke-keys"
Oct 31 16:47:06.756869 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.756858    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="063ced0c-f028-4527-a9d4-b398c2ac4a2e" containerName="revoke-keys"
Oct 31 16:47:06.760052 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.760033    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.792301 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.792268    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5zk\" (UniqueName: \"kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.792441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.792326    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.792441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.792397    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.811737 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.811704    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"]
Oct 31 16:47:06.855908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.855873    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:47:06.855908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.855873    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:47:06.857223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.857204    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:47:06.857359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.857343    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:47:06.893425 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.893395    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5zk\" (UniqueName: \"kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.893614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.893441    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.893614 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.893497    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.894248 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.894225    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.895852 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.895833    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:06.947108 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:06.947081    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5zk\" (UniqueName: \"kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk\") pod \"maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:07.043106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.043022    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"]
Oct 31 16:47:07.045148 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.045121    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9bbbdf-hnxnp"]
Oct 31 16:47:07.073231 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.073206    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:07.444525 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.444435    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"]
Oct 31 16:47:07.450633 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:47:07.450603    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ffb855_1b54_48e6_a7a2_a990981d79fd.slice/crio-a2270da116c9838d3f3ffec3e4328b62b303a0809d65af4169996d378c0a430e WatchSource:0}: Error finding container a2270da116c9838d3f3ffec3e4328b62b303a0809d65af4169996d378c0a430e: Status 404 returned error can't find the container with id a2270da116c9838d3f3ffec3e4328b62b303a0809d65af4169996d378c0a430e
Oct 31 16:47:07.829286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.829246    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58" event={"ID":"70ffb855-1b54-48e6-a7a2-a990981d79fd","Type":"ContainerStarted","Data":"2c7e50a32b19048985c8f751010aa64fbcc333fae139bae4e9a582602067b893"}
Oct 31 16:47:07.829686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.829305    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58" event={"ID":"70ffb855-1b54-48e6-a7a2-a990981d79fd","Type":"ContainerStarted","Data":"a2270da116c9838d3f3ffec3e4328b62b303a0809d65af4169996d378c0a430e"}
Oct 31 16:47:07.938066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:07.938005    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58" podStartSLOduration=1.937990358 podStartE2EDuration="1.937990358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:47:07.936428536 +0000 UTC m=+2671.636357847" watchObservedRunningTime="2026-10-31 16:47:07.937990358 +0000 UTC m=+2671.637919675"
Oct 31 16:47:08.438276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:08.438241    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4b23cd-846c-4b6d-b2bd-b89714bed1e5" path="/var/lib/kubelet/pods/0a4b23cd-846c-4b6d-b2bd-b89714bed1e5/volumes"
Oct 31 16:47:08.836299 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:08.836265    4519 generic.go:356] "Generic (PLEG): container finished" podID="70ffb855-1b54-48e6-a7a2-a990981d79fd" containerID="2c7e50a32b19048985c8f751010aa64fbcc333fae139bae4e9a582602067b893" exitCode=0
Oct 31 16:47:08.836736 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:08.836351    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58" event={"ID":"70ffb855-1b54-48e6-a7a2-a990981d79fd","Type":"ContainerDied","Data":"2c7e50a32b19048985c8f751010aa64fbcc333fae139bae4e9a582602067b893"}
Oct 31 16:47:09.981650 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:09.981627    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:10.023307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.023276    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca\") pod \"70ffb855-1b54-48e6-a7a2-a990981d79fd\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") "
Oct 31 16:47:10.023502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.023315    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk\" (UniqueName: \"kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk\") pod \"70ffb855-1b54-48e6-a7a2-a990981d79fd\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") "
Oct 31 16:47:10.023502 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.023403    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca\") pod \"70ffb855-1b54-48e6-a7a2-a990981d79fd\" (UID: \"70ffb855-1b54-48e6-a7a2-a990981d79fd\") "
Oct 31 16:47:10.023731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.023704    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca" pod "70ffb855-1b54-48e6-a7a2-a990981d79fd" (UID: "70ffb855-1b54-48e6-a7a2-a990981d79fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:10.025585 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.025563    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk" pod "70ffb855-1b54-48e6-a7a2-a990981d79fd" (UID: "70ffb855-1b54-48e6-a7a2-a990981d79fd"). InnerVolumeSpecName "kube-api-access-xm5zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:10.025662 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.025589    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca" pod "70ffb855-1b54-48e6-a7a2-a990981d79fd" (UID: "70ffb855-1b54-48e6-a7a2-a990981d79fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:47:10.124957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.124874    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:10.124957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.124903    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70ffb855-1b54-48e6-a7a2-a990981d79fd-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:10.124957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.124914    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5zk\" (UniqueName: \"kubernetes.io/projected/70ffb855-1b54-48e6-a7a2-a990981d79fd-kube-api-access-xm5zk\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:10.849078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.849045    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"
Oct 31 16:47:10.849241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.849082    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58" event={"ID":"70ffb855-1b54-48e6-a7a2-a990981d79fd","Type":"ContainerDied","Data":"a2270da116c9838d3f3ffec3e4328b62b303a0809d65af4169996d378c0a430e"}
Oct 31 16:47:10.849241 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:10.849133    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2270da116c9838d3f3ffec3e4328b62b303a0809d65af4169996d378c0a430e"
Oct 31 16:47:15.042982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:15.042941    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"]
Oct 31 16:47:15.090137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:15.090105    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbc374d-lqwwf"]
Oct 31 16:47:16.044982 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.044912    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"]
Oct 31 16:47:16.090992 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.090956    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"]
Oct 31 16:47:16.093136 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.093110    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"]
Oct 31 16:47:16.094906 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.094891    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04403fee-xwvqr"]
Oct 31 16:47:16.096660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.096636    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f353f207-nxx4g"]
Oct 31 16:47:16.137693 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.137664    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd97a17-zz9zx"]
Oct 31 16:47:16.437341 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.437268    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e5fcce-5a1e-459e-a82e-07ed0aad66db" path="/var/lib/kubelet/pods/16e5fcce-5a1e-459e-a82e-07ed0aad66db/volumes"
Oct 31 16:47:16.437684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.437670    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97177123-adf9-462a-b281-af0a610c259f" path="/var/lib/kubelet/pods/97177123-adf9-462a-b281-af0a610c259f/volumes"
Oct 31 16:47:16.438057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.438044    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f45d47-be0b-4e04-aef5-0dbc9639c5f8" path="/var/lib/kubelet/pods/b2f45d47-be0b-4e04-aef5-0dbc9639c5f8/volumes"
Oct 31 16:47:16.438689 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:16.438664    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0a2644-e5c8-4304-a435-54784705a527" path="/var/lib/kubelet/pods/ec0a2644-e5c8-4304-a435-54784705a527/volumes"
Oct 31 16:47:24.003993 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:47:24.003948    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:47:30.778156 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:30.778116    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"]
Oct 31 16:47:30.825173 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:30.825138    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-66db1c-kbk58"]
Oct 31 16:47:32.045176 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.045138    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"]
Oct 31 16:47:32.093567 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.093538    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"]
Oct 31 16:47:32.095484 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.095441    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55df6fe1-pphs5"]
Oct 31 16:47:32.097139 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.097117    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d5948d8-88nzg"]
Oct 31 16:47:32.438354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.438278    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449026bc-7a74-4640-b65f-564a85c04705" path="/var/lib/kubelet/pods/449026bc-7a74-4640-b65f-564a85c04705/volumes"
Oct 31 16:47:32.438715 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.438701    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5596b388-86a8-4ba7-86d2-11f50a01d63f" path="/var/lib/kubelet/pods/5596b388-86a8-4ba7-86d2-11f50a01d63f/volumes"
Oct 31 16:47:32.439069 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:32.439057    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ffb855-1b54-48e6-a7a2-a990981d79fd" path="/var/lib/kubelet/pods/70ffb855-1b54-48e6-a7a2-a990981d79fd/volumes"
Oct 31 16:47:34.204533 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:34.204497    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:47:38.200864 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.200832    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"]
Oct 31 16:47:38.201322 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.201307    4519 cpu_manager.go:414] "containerMap: removing container" podUID="70ffb855-1b54-48e6-a7a2-a990981d79fd" containerName="revoke-keys"
Oct 31 16:47:38.201371 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.201324    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ffb855-1b54-48e6-a7a2-a990981d79fd" containerName="revoke-keys"
Oct 31 16:47:38.201430 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.201420    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ffb855-1b54-48e6-a7a2-a990981d79fd" containerName="revoke-keys"
Oct 31 16:47:38.204716 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.204695    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.239074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.239046    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5295v\" (UniqueName: \"kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.239203 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.239183    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.239251 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.239239    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.257183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.257156    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"]
Oct 31 16:47:38.301790 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.301764    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:47:38.302057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.302041    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:47:38.302057 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.302050    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:47:38.302169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.302151    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:47:38.340194 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.340129    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.340362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.340195    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.340362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.340245    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5295v\" (UniqueName: \"kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.340853 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.340832    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.342442 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.342424    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.394349 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.394314    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5295v\" (UniqueName: \"kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v\") pod \"maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.517170 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.517140    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:47:38.889496 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:38.889392    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"]
Oct 31 16:47:38.896466 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:47:38.896413    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb423fd7_366b_4ffb_87ef_ed2f77eea17f.slice/crio-221c7891b247e493b5bf553a18f5289b0cea51d90c8a11f5d95e1cf480d97431 WatchSource:0}: Error finding container 221c7891b247e493b5bf553a18f5289b0cea51d90c8a11f5d95e1cf480d97431: Status 404 returned error can't find the container with id 221c7891b247e493b5bf553a18f5289b0cea51d90c8a11f5d95e1cf480d97431
Oct 31 16:47:39.028255 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:39.028222    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl" event={"ID":"cb423fd7-366b-4ffb-87ef-ed2f77eea17f","Type":"ContainerStarted","Data":"221c7891b247e493b5bf553a18f5289b0cea51d90c8a11f5d95e1cf480d97431"}
Oct 31 16:47:39.257253 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:39.257223    4519 scope.go:122] "RemoveContainer" containerID="42d8e2ce3ab93080af40fcb69b2536a7adf78222b2cb506553fc804c2906cfef"
Oct 31 16:47:39.266480 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:39.266443    4519 scope.go:122] "RemoveContainer" containerID="71180388ef09bf5b0d798b33992b31082dc5df264d264b3946905fabfa2b06af"
Oct 31 16:47:39.274359 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:39.274330    4519 scope.go:122] "RemoveContainer" containerID="1abc04b566bb96bc591e183aefbc5a14c0f6e5f49a3b91fee4ac484a6915a18d"
Oct 31 16:47:40.035930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:40.035895    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl" event={"ID":"cb423fd7-366b-4ffb-87ef-ed2f77eea17f","Type":"ContainerStarted","Data":"9a1d32ba898285c58a77e63e41a3417729b254db761c3ee641360baabb952b03"}
Oct 31 16:47:40.143786 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:40.143720    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl" podStartSLOduration=2.143701419 podStartE2EDuration="2.143701419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:47:40.142427232 +0000 UTC m=+2703.842356547" watchObservedRunningTime="2026-10-31 16:47:40.143701419 +0000 UTC m=+2703.843630734"
Oct 31 16:47:42.662817 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:42.662782    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"]
Oct 31 16:47:54.003674 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:47:54.003634    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:47:57.683651 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:57.683611    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main" containerID="cri-o://a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db" gracePeriod=45
Oct 31 16:47:57.936225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:57.936175    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:47:58.044647 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.044451    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2\" (UniqueName: \"kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2\") pod \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") "
Oct 31 16:47:58.044838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.044700    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache\") pod \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") "
Oct 31 16:47:58.044838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.044796    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir\") pod \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") "
Oct 31 16:47:58.044838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.044827    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm\") pod \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") "
Oct 31 16:47:58.045040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.044875    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home\") pod \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") "
Oct 31 16:47:58.045040 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.044903    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs\") pod \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\" (UID: \"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a\") "
Oct 31 16:47:58.045693 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.045621    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home" pod "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" (UID: "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.045914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.045860    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir" pod "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" (UID: "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.046145 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.046114    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache" pod "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" (UID: "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.047386 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.047348    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2" pod "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" (UID: "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a"). InnerVolumeSpecName "kube-api-access-mjpq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:58.047616 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.047585    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm" pod "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" (UID: "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.049028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.048822    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs" pod "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" (UID: "16d861ab-eb14-4b79-a9a4-5b90da2f2b3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:47:58.133840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.133805    4519 generic.go:356] "Generic (PLEG): container finished" podID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerID="a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db" exitCode=0
Oct 31 16:47:58.134019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.133875    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"
Oct 31 16:47:58.134019 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.133987    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" event={"ID":"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a","Type":"ContainerDied","Data":"a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db"}
Oct 31 16:47:58.134137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.134026    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5" event={"ID":"16d861ab-eb14-4b79-a9a4-5b90da2f2b3a","Type":"ContainerDied","Data":"392a322db379175597e61e490241f004dd398d456fa370de2b7258678e5a7387"}
Oct 31 16:47:58.134137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.134056    4519 scope.go:122] "RemoveContainer" containerID="a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db"
Oct 31 16:47:58.144448 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.144429    4519 scope.go:122] "RemoveContainer" containerID="a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db"
Oct 31 16:47:58.144773 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:47:58.144755    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db\": container with ID starting with a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db not found: ID does not exist" containerID="a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db"
Oct 31 16:47:58.144847 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.144781    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db"} err="failed to get container status \"a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db\": rpc error: code = NotFound desc = could not find container \"a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db\": container with ID starting with a61c584825618fb4e75b469c7b7740e5bfa36db3f51a7815f442ebfe0a1969db not found: ID does not exist"
Oct 31 16:47:58.146279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.146258    4519 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tmp-dir\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.146354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.146281    4519 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-dshm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.146354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.146289    4519 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-home\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.146354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.146297    4519 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-tls-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.146354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.146306    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpq2\" (UniqueName: \"kubernetes.io/projected/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-kube-api-access-mjpq2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.146354 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.146315    4519 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a-model-cache\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.289407 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.289374    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"]
Oct 31 16:47:58.293300 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.293276    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-2-w3-f7821a-kserve-79dcddc7d-ggnf5"]
Oct 31 16:47:58.438104 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:47:58.438069    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" path="/var/lib/kubelet/pods/16d861ab-eb14-4b79-a9a4-5b90da2f2b3a/volumes"
Oct 31 16:48:10.200309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:10.200276    4519 generic.go:356] "Generic (PLEG): container finished" podID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerID="9a1d32ba898285c58a77e63e41a3417729b254db761c3ee641360baabb952b03" exitCode=28
Oct 31 16:48:10.200828 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:10.200354    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl" event={"ID":"cb423fd7-366b-4ffb-87ef-ed2f77eea17f","Type":"ContainerDied","Data":"9a1d32ba898285c58a77e63e41a3417729b254db761c3ee641360baabb952b03"}
Oct 31 16:48:10.200921 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:10.200832    4519 scope.go:122] "RemoveContainer" containerID="9a1d32ba898285c58a77e63e41a3417729b254db761c3ee641360baabb952b03"
Oct 31 16:48:11.207227 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:11.207195    4519 generic.go:356] "Generic (PLEG): container finished" podID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerID="d318e4c9c9d840feaf74ace1bd9fb73ebea5fb84f3a96e793b1d2180e7819b1b" exitCode=0
Oct 31 16:48:11.207724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:11.207267    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl" event={"ID":"cb423fd7-366b-4ffb-87ef-ed2f77eea17f","Type":"ContainerDied","Data":"d318e4c9c9d840feaf74ace1bd9fb73ebea5fb84f3a96e793b1d2180e7819b1b"}
Oct 31 16:48:11.207724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:11.207329    4519 scope.go:122] "RemoveContainer" containerID="9a1d32ba898285c58a77e63e41a3417729b254db761c3ee641360baabb952b03"
Oct 31 16:48:12.355362 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.355338    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:48:12.489327 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.489255    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v\" (UniqueName: \"kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v\") pod \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") "
Oct 31 16:48:12.489527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.489477    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca\") pod \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") "
Oct 31 16:48:12.489527 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.489506    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca\") pod \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\" (UID: \"cb423fd7-366b-4ffb-87ef-ed2f77eea17f\") "
Oct 31 16:48:12.489931 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.489905    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca" pod "cb423fd7-366b-4ffb-87ef-ed2f77eea17f" (UID: "cb423fd7-366b-4ffb-87ef-ed2f77eea17f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:12.491414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.491392    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v" pod "cb423fd7-366b-4ffb-87ef-ed2f77eea17f" (UID: "cb423fd7-366b-4ffb-87ef-ed2f77eea17f"). InnerVolumeSpecName "kube-api-access-5295v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:12.491609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.491590    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca" pod "cb423fd7-366b-4ffb-87ef-ed2f77eea17f" (UID: "cb423fd7-366b-4ffb-87ef-ed2f77eea17f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:48:12.590858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.590827    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5295v\" (UniqueName: \"kubernetes.io/projected/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-kube-api-access-5295v\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:12.590858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.590856    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:12.591047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:12.590867    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb423fd7-366b-4ffb-87ef-ed2f77eea17f-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:13.221490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:13.221442    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl" event={"ID":"cb423fd7-366b-4ffb-87ef-ed2f77eea17f","Type":"ContainerDied","Data":"221c7891b247e493b5bf553a18f5289b0cea51d90c8a11f5d95e1cf480d97431"}
Oct 31 16:48:13.221490 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:13.221494    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"
Oct 31 16:48:13.221705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:13.221512    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221c7891b247e493b5bf553a18f5289b0cea51d90c8a11f5d95e1cf480d97431"
Oct 31 16:48:17.678622 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:17.678541    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"]
Oct 31 16:48:17.679064 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:17.678727    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy" containerID="cri-o://8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373" gracePeriod=30
Oct 31 16:48:17.725923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:17.725889    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"]
Oct 31 16:48:17.726117 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:17.726092    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy" containerID="cri-o://2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05" gracePeriod=30
Oct 31 16:48:19.779870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:19.779828    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"]
Oct 31 16:48:20.715032 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715002    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"]
Oct 31 16:48:20.715474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715441    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerName="revoke-keys"
Oct 31 16:48:20.715474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715468    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerName="revoke-keys"
Oct 31 16:48:20.715569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715490    4519 cpu_manager.go:414] "containerMap: removing container" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerName="revoke-keys"
Oct 31 16:48:20.715569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715496    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerName="revoke-keys"
Oct 31 16:48:20.715569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715510    4519 cpu_manager.go:414] "containerMap: removing container" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main"
Oct 31 16:48:20.715569 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715516    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main"
Oct 31 16:48:20.715700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715584    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerName="revoke-keys"
Oct 31 16:48:20.715700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715595    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" containerName="revoke-keys"
Oct 31 16:48:20.715700 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.715604    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d861ab-eb14-4b79-a9a4-5b90da2f2b3a" containerName="main"
Oct 31 16:48:20.719062 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.719045    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.769694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.769667    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.769923 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.769893    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.770080 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.770060    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.771812 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.771785    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"]
Oct 31 16:48:20.777731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.777705    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:48:20.777837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.777710    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:48:20.777837 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.777770    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:48:20.777970 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.777953    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:48:20.870962 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.870926    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.871373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.871022    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.871373 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.871051    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.871606 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.871578    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.873555 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.873526    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:20.924696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:20.924671    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d\") pod \"maas-api-revoke-sub-8969fb08-qchj2\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:21.031833 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:21.031755    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:21.402656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:21.402579    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"]
Oct 31 16:48:21.409923 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:48:21.409890    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88adb615_4145_4517_94de_2d27daa62814.slice/crio-6b576ea3ccea005860457cd6e40fda165d2a622629433bc687d477adb0b15118 WatchSource:0}: Error finding container 6b576ea3ccea005860457cd6e40fda165d2a622629433bc687d477adb0b15118: Status 404 returned error can't find the container with id 6b576ea3ccea005860457cd6e40fda165d2a622629433bc687d477adb0b15118
Oct 31 16:48:22.271006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:22.270960    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2" event={"ID":"88adb615-4145-4517-94de-2d27daa62814","Type":"ContainerStarted","Data":"0c3a6dd3c7e09ee3ffbc70c1ea3d638f3b42fdaef1998bdc521b2b1aea21cd96"}
Oct 31 16:48:22.271413 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:22.271025    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2" event={"ID":"88adb615-4145-4517-94de-2d27daa62814","Type":"ContainerStarted","Data":"6b576ea3ccea005860457cd6e40fda165d2a622629433bc687d477adb0b15118"}
Oct 31 16:48:22.379609 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:22.379558    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2" podStartSLOduration=2.3795447149999998 podStartE2EDuration="2.379544715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:48:22.377561933 +0000 UTC m=+2746.077491243" watchObservedRunningTime="2026-10-31 16:48:22.379544715 +0000 UTC m=+2746.079474026"
Oct 31 16:48:23.000089 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.000060    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:48:23.002694 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.002676    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:48:23.093420 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093384    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093428    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093475    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.093603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093509    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093550    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.093603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093599    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093619    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093653    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093699    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093728    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093748    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093796    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093822    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093860    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data\") pod \"af17d887-3bc7-4077-a2dc-755c296a3b12\" (UID: \"af17d887-3bc7-4077-a2dc-755c296a3b12\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093881    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.093914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093907    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.094698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093931    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.094698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.093964    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket\") pod \"1e91534f-e6fa-4967-a948-9fd10cd089a8\" (UID: \"1e91534f-e6fa-4967-a948-9fd10cd089a8\") "
Oct 31 16:48:23.094698 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.094488    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.094893 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.094702    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.094965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.094944    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.095126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.095105    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.095126 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.095118    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.095247 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.095186    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.096047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.095371    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.096047 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.095572    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.099250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.097591    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.099250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.097864    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.099250 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.098544    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:48:23.099952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.099924    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.100144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.100122    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "kube-api-access-ghlcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.100223 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.100128    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.100286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.100238    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.100346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.100315    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo" pod "af17d887-3bc7-4077-a2dc-755c296a3b12" (UID: "af17d887-3bc7-4077-a2dc-755c296a3b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:48:23.100346 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.100335    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "kube-api-access-5nvl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.101329 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.101307    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token" pod "1e91534f-e6fa-4967-a948-9fd10cd089a8" (UID: "1e91534f-e6fa-4967-a948-9fd10cd089a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.195493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195447    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195490    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195493 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195501    4519 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195510    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghlcz\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-kube-api-access-ghlcz\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195519    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195529    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195537    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195546    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvl4\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-kube-api-access-5nvl4\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195554    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195562    4519 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-envoy\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195570    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af17d887-3bc7-4077-a2dc-755c296a3b12-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195578    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195586    4519 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-workload-certs\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195594    4519 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af17d887-3bc7-4077-a2dc-755c296a3b12-istio-data\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195602    4519 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e91534f-e6fa-4967-a948-9fd10cd089a8-istiod-ca-cert\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195609    4519 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-podinfo\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195617    4519 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e91534f-e6fa-4967-a948-9fd10cd089a8-istio-token\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.195731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.195624    4519 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e91534f-e6fa-4967-a948-9fd10cd089a8-credential-socket\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.280886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.280791    4519 generic.go:356] "Generic (PLEG): container finished" podID="88adb615-4145-4517-94de-2d27daa62814" containerID="0c3a6dd3c7e09ee3ffbc70c1ea3d638f3b42fdaef1998bdc521b2b1aea21cd96" exitCode=0
Oct 31 16:48:23.280886 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.280866    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2" event={"ID":"88adb615-4145-4517-94de-2d27daa62814","Type":"ContainerDied","Data":"0c3a6dd3c7e09ee3ffbc70c1ea3d638f3b42fdaef1998bdc521b2b1aea21cd96"}
Oct 31 16:48:23.282286 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.282260    4519 generic.go:356] "Generic (PLEG): container finished" podID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerID="2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05" exitCode=0
Oct 31 16:48:23.282429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.282304    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"
Oct 31 16:48:23.282429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.282342    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" event={"ID":"1e91534f-e6fa-4967-a948-9fd10cd089a8","Type":"ContainerDied","Data":"2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05"}
Oct 31 16:48:23.282429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.282397    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj" event={"ID":"1e91534f-e6fa-4967-a948-9fd10cd089a8","Type":"ContainerDied","Data":"e560e5aa5f77f0893dcdeac2df95bbf1cd19b4c86b4a138a0b06ae38099a0b35"}
Oct 31 16:48:23.282429 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.282420    4519 scope.go:122] "RemoveContainer" containerID="2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05"
Oct 31 16:48:23.283903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.283884    4519 generic.go:356] "Generic (PLEG): container finished" podID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerID="8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373" exitCode=0
Oct 31 16:48:23.284016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.283968    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" event={"ID":"af17d887-3bc7-4077-a2dc-755c296a3b12","Type":"ContainerDied","Data":"8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373"}
Oct 31 16:48:23.284016 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.283985    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"
Oct 31 16:48:23.284127 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.284066    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt" event={"ID":"af17d887-3bc7-4077-a2dc-755c296a3b12","Type":"ContainerDied","Data":"f6a62afb6bfbe2e713a94fd6e0de048690eadfd4eed478f622889f5a7b6ce6f6"}
Oct 31 16:48:23.294891 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.294870    4519 scope.go:122] "RemoveContainer" containerID="2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05"
Oct 31 16:48:23.295174 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:48:23.295151    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05\": container with ID starting with 2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05 not found: ID does not exist" containerID="2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05"
Oct 31 16:48:23.295268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.295189    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05"} err="failed to get container status \"2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05\": rpc error: code = NotFound desc = could not find container \"2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05\": container with ID starting with 2d55dd0d6b14a11c471b3121a28cf0bb86f61ce8735071b8363ac85ce644ef05 not found: ID does not exist"
Oct 31 16:48:23.295268 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.295213    4519 scope.go:122] "RemoveContainer" containerID="8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373"
Oct 31 16:48:23.304632 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.304612    4519 scope.go:122] "RemoveContainer" containerID="8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373"
Oct 31 16:48:23.304891 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:48:23.304874    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373\": container with ID starting with 8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373 not found: ID does not exist" containerID="8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373"
Oct 31 16:48:23.304933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.304898    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373"} err="failed to get container status \"8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373\": rpc error: code = NotFound desc = could not find container \"8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373\": container with ID starting with 8648f866363a9840781878b41547fad1de7eca1ed0fcd1f6a580c84a74077373 not found: ID does not exist"
Oct 31 16:48:23.492880 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.492846    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"]
Oct 31 16:48:23.497119 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.497094    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cpsqt"]
Oct 31 16:48:23.599826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.599736    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"]
Oct 31 16:48:23.603195 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:23.603169    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qr6gj"]
Oct 31 16:48:24.009814 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:48:24.006663    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:48:24.428957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.428933    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:24.437644 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.437618    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" path="/var/lib/kubelet/pods/1e91534f-e6fa-4967-a948-9fd10cd089a8/volumes"
Oct 31 16:48:24.438101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.438087    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" path="/var/lib/kubelet/pods/af17d887-3bc7-4077-a2dc-755c296a3b12/volumes"
Oct 31 16:48:24.508543 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.508512    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca\") pod \"88adb615-4145-4517-94de-2d27daa62814\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") "
Oct 31 16:48:24.508714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.508557    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca\") pod \"88adb615-4145-4517-94de-2d27daa62814\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") "
Oct 31 16:48:24.508714 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.508692    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d\") pod \"88adb615-4145-4517-94de-2d27daa62814\" (UID: \"88adb615-4145-4517-94de-2d27daa62814\") "
Oct 31 16:48:24.508965 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.508943    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca" pod "88adb615-4145-4517-94de-2d27daa62814" (UID: "88adb615-4145-4517-94de-2d27daa62814"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:24.509122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.509104    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88adb615-4145-4517-94de-2d27daa62814-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:24.510750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.510729    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca" pod "88adb615-4145-4517-94de-2d27daa62814" (UID: "88adb615-4145-4517-94de-2d27daa62814"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:48:24.510844 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.510825    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d" pod "88adb615-4145-4517-94de-2d27daa62814" (UID: "88adb615-4145-4517-94de-2d27daa62814"). InnerVolumeSpecName "kube-api-access-4r45d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:24.609952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.609878    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88adb615-4145-4517-94de-2d27daa62814-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:24.609952 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.609905    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/88adb615-4145-4517-94de-2d27daa62814-kube-api-access-4r45d\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:24.780368 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:24.780326    4519 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2" podUID="585965f9-fd7f-4617-bac8-fa97dec765e1" containerName="payload-processing" containerID="cri-o://4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80" gracePeriod=25
Oct 31 16:48:25.033522 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.033498    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:48:25.116060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.116035    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume\") pod \"585965f9-fd7f-4617-bac8-fa97dec765e1\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") "
Oct 31 16:48:25.116214 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.116081    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5\" (UniqueName: \"kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5\") pod \"585965f9-fd7f-4617-bac8-fa97dec765e1\" (UID: \"585965f9-fd7f-4617-bac8-fa97dec765e1\") "
Oct 31 16:48:25.116507 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.116482    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume" pod "585965f9-fd7f-4617-bac8-fa97dec765e1" (UID: "585965f9-fd7f-4617-bac8-fa97dec765e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:25.118116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.118097    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5" pod "585965f9-fd7f-4617-bac8-fa97dec765e1" (UID: "585965f9-fd7f-4617-bac8-fa97dec765e1"). InnerVolumeSpecName "kube-api-access-h6lc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:25.217838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.217802    4519 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/585965f9-fd7f-4617-bac8-fa97dec765e1-plugins-config-volume\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:25.217838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.217835    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6lc5\" (UniqueName: \"kubernetes.io/projected/585965f9-fd7f-4617-bac8-fa97dec765e1-kube-api-access-h6lc5\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:25.300128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.300099    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"
Oct 31 16:48:25.300128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.300116    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2" event={"ID":"88adb615-4145-4517-94de-2d27daa62814","Type":"ContainerDied","Data":"6b576ea3ccea005860457cd6e40fda165d2a622629433bc687d477adb0b15118"}
Oct 31 16:48:25.300356 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.300188    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b576ea3ccea005860457cd6e40fda165d2a622629433bc687d477adb0b15118"
Oct 31 16:48:25.301545 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.301515    4519 generic.go:356] "Generic (PLEG): container finished" podID="585965f9-fd7f-4617-bac8-fa97dec765e1" containerID="4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80" exitCode=0
Oct 31 16:48:25.301660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.301549    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"
Oct 31 16:48:25.301660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.301594    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2" event={"ID":"585965f9-fd7f-4617-bac8-fa97dec765e1","Type":"ContainerDied","Data":"4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80"}
Oct 31 16:48:25.301660 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.301636    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2" event={"ID":"585965f9-fd7f-4617-bac8-fa97dec765e1","Type":"ContainerDied","Data":"81909a2b00e51eaa9337a7c7eb801616165d8d14509f2c0dcbe4afd386bb994b"}
Oct 31 16:48:25.301876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.301662    4519 scope.go:122] "RemoveContainer" containerID="4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80"
Oct 31 16:48:25.312661 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.312592    4519 scope.go:122] "RemoveContainer" containerID="4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80"
Oct 31 16:48:25.312865 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:48:25.312848    4519 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80\": container with ID starting with 4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80 not found: ID does not exist" containerID="4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80"
Oct 31 16:48:25.312910 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.312873    4519 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80"} err="failed to get container status \"4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80\": rpc error: code = NotFound desc = could not find container \"4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80\": container with ID starting with 4ca37932d34b2a1f8f6f31c5f8445390c30aedfa233b523add3ef3a22415ff80 not found: ID does not exist"
Oct 31 16:48:25.458267 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.458239    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"]
Oct 31 16:48:25.460592 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:25.460558    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f7821a-7bd5cd9755-dl8q2"]
Oct 31 16:48:26.437986 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:26.437955    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585965f9-fd7f-4617-bac8-fa97dec765e1" path="/var/lib/kubelet/pods/585965f9-fd7f-4617-bac8-fa97dec765e1/volumes"
Oct 31 16:48:32.802745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:32.802706    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"]
Oct 31 16:48:32.849919 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:32.849886    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f7821a-ptmkl"]
Oct 31 16:48:34.437866 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:34.437835    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb423fd7-366b-4ffb-87ef-ed2f77eea17f" path="/var/lib/kubelet/pods/cb423fd7-366b-4ffb-87ef-ed2f77eea17f/volumes"
Oct 31 16:48:35.413573 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:35.413540    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:48:39.304422 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.304391    4519 scope.go:122] "RemoveContainer" containerID="57b9643bc1a0ece95657e9bbd8acd9a600206ba145f031d6c05f41adc5f85496"
Oct 31 16:48:39.313169 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.313137    4519 scope.go:122] "RemoveContainer" containerID="6110b7773f0adafc450cf57913474e7e703535ea1fec62f83014554d66bc07f2"
Oct 31 16:48:39.321210 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.321189    4519 scope.go:122] "RemoveContainer" containerID="1de463b01b98c529ea776f6cd6c5a89ffbfbf192be7257755a8eed8b6abd886a"
Oct 31 16:48:39.329854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.329820    4519 scope.go:122] "RemoveContainer" containerID="fce3aa35e7414b9bb5671c094103c83e4d386870e365b3ba081c27cf542ce8fc"
Oct 31 16:48:39.337913 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.337893    4519 scope.go:122] "RemoveContainer" containerID="27088589cc4cd21257e7add52497e0512083141ecffb4a18cce5280a66bf8222"
Oct 31 16:48:39.346030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.346009    4519 scope.go:122] "RemoveContainer" containerID="8b071132c957e626e6d234efd4a7f99b465884f5cf378ba54d48ac18a4d10a16"
Oct 31 16:48:39.353534 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.353511    4519 scope.go:122] "RemoveContainer" containerID="6a131df04fcfa0f6c603f709ec7b0131ae00a6c410d676f4cf4e6669189fc0c1"
Oct 31 16:48:39.364199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.364168    4519 scope.go:122] "RemoveContainer" containerID="4160542ed820c31f1771625fbca58ad9ec77b34c60cea5583d5c3091b71eabef"
Oct 31 16:48:39.371724 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:48:39.371703    4519 scope.go:122] "RemoveContainer" containerID="4ea787abd9d86579253bf54e5442a3fd41ba7e2691115f31c7cf18e872f9d307"
Oct 31 16:48:54.003093 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:48:54.003055    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:49:18.164981 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.164944    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"]
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165342    4519 cpu_manager.go:414] "containerMap: removing container" podUID="88adb615-4145-4517-94de-2d27daa62814" containerName="revoke-keys"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165353    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88adb615-4145-4517-94de-2d27daa62814" containerName="revoke-keys"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165372    4519 cpu_manager.go:414] "containerMap: removing container" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165378    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165387    4519 cpu_manager.go:414] "containerMap: removing container" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165393    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165404    4519 cpu_manager.go:414] "containerMap: removing container" podUID="585965f9-fd7f-4617-bac8-fa97dec765e1" containerName="payload-processing"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165409    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585965f9-fd7f-4617-bac8-fa97dec765e1" containerName="payload-processing"
Oct 31 16:49:18.165504 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165500    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="88adb615-4145-4517-94de-2d27daa62814" containerName="revoke-keys"
Oct 31 16:49:18.165805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165512    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e91534f-e6fa-4967-a948-9fd10cd089a8" containerName="istio-proxy"
Oct 31 16:49:18.165805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165518    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="af17d887-3bc7-4077-a2dc-755c296a3b12" containerName="istio-proxy"
Oct 31 16:49:18.165805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.165525    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="585965f9-fd7f-4617-bac8-fa97dec765e1" containerName="payload-processing"
Oct 31 16:49:18.168838 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.168820    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.250109 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.250077    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"]
Oct 31 16:49:18.319074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319040    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24f8e20-1030-4e69-96c6-a4a20a579947-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319102    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319226    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319264 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319260    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319381 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319333    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319417 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319388    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319420    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhcp\" (UniqueName: \"kubernetes.io/projected/f24f8e20-1030-4e69-96c6-a4a20a579947-kube-api-access-qlhcp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319449    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.319554 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.319517    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.420735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420658    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.420735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420693    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhcp\" (UniqueName: \"kubernetes.io/projected/f24f8e20-1030-4e69-96c6-a4a20a579947-kube-api-access-qlhcp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.420735 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420711    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420740    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420763    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24f8e20-1030-4e69-96c6-a4a20a579947-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420803    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420827    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420847    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421045 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.420899    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421388 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.421193    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421474 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.421439    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.421474    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.421504    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.421675 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.421657    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24f8e20-1030-4e69-96c6-a4a20a579947-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.423090 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.423069    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.423238 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.423219    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.474489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.474436    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24f8e20-1030-4e69-96c6-a4a20a579947-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.474638 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.474535    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhcp\" (UniqueName: \"kubernetes.io/projected/f24f8e20-1030-4e69-96c6-a4a20a579947-kube-api-access-qlhcp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5tkrq\" (UID: \"f24f8e20-1030-4e69-96c6-a4a20a579947\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.481414 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.481389    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:18.902826 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.902781    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"]
Oct 31 16:49:18.908972 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:49:18.908942    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24f8e20_1030_4e69_96c6_a4a20a579947.slice/crio-e52b734b37a8f812b346d2ffa02905e6ed2771bddf933b223b045602fc3b603c WatchSource:0}: Error finding container e52b734b37a8f812b346d2ffa02905e6ed2771bddf933b223b045602fc3b603c: Status 404 returned error can't find the container with id e52b734b37a8f812b346d2ffa02905e6ed2771bddf933b223b045602fc3b603c
Oct 31 16:49:18.911229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.911193    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:18.911309 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.911275    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:18.911352 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:18.911317    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:19.597141 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:19.597102    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" event={"ID":"f24f8e20-1030-4e69-96c6-a4a20a579947","Type":"ContainerStarted","Data":"d80390a694b72e24ff02822e193600d6bc869344000a396dc2679e55ebe6cab8"}
Oct 31 16:49:19.597572 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:19.597163    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" event={"ID":"f24f8e20-1030-4e69-96c6-a4a20a579947","Type":"ContainerStarted","Data":"e52b734b37a8f812b346d2ffa02905e6ed2771bddf933b223b045602fc3b603c"}
Oct 31 16:49:19.806175 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:19.806103    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podStartSLOduration=1.806089696 podStartE2EDuration="1.806089696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:49:19.758269288 +0000 UTC m=+2803.458198603" watchObservedRunningTime="2026-10-31 16:49:19.806089696 +0000 UTC m=+2803.506019007"
Oct 31 16:49:20.486479 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:20.485514    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:20.490588 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:20.490555    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 31 16:49:20.490745 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:20.490613    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 31 16:49:21.481938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:21.481899    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 31 16:49:21.482552 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:21.482518    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 31 16:49:22.490481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:22.486078    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 31 16:49:22.490481 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:22.486140    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 31 16:49:23.482927 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:23.482888    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 31 16:49:23.483239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:23.483209    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 31 16:49:24.017042 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:49:24.017006    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:49:24.484930 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:24.484892    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 31 16:49:24.485283 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:24.485243    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 31 16:49:25.482768 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:25.482722    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Oct 31 16:49:25.483167 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:25.482806    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Oct 31 16:49:27.203778 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:27.203741    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:27.204288 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:27.204087    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:27.205182 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:49:27.205155    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq"
Oct 31 16:49:54.006317 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:49:54.006271    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:50:11.011731 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:11.011690    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:50:13.245665 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:13.245616    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:13.246078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:13.245692    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:24.004308 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:50:24.004269    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:50:32.466470 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:32.466421    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:50:33.718508 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.718471    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"]
Oct 31 16:50:33.722601 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.722579    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802225 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802182    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802243    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1158-9e56-4558-b7f5-2272af25e327-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802331    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802352    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802376    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6kx\" (UniqueName: \"kubernetes.io/projected/6cdf1158-9e56-4558-b7f5-2272af25e327-kube-api-access-rq6kx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802437 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802399    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802493    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802612    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.802781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.802644    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903636    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903682 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903675    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903726    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903771    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1158-9e56-4558-b7f5-2272af25e327-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903820    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903845    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903877    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6kx\" (UniqueName: \"kubernetes.io/projected/6cdf1158-9e56-4558-b7f5-2272af25e327-kube-api-access-rq6kx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903911    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.903968 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.903944    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.904406 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.904118    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.904746 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.904689    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf1158-9e56-4558-b7f5-2272af25e327-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.905028 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.905001    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.905174 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.905153    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.905258 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.905195    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.906873 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.906846    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.910478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.908135    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.911266 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.911192    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"]
Oct 31 16:50:33.958750 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.958714    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdf1158-9e56-4558-b7f5-2272af25e327-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:33.958903 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:33.958771    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6kx\" (UniqueName: \"kubernetes.io/projected/6cdf1158-9e56-4558-b7f5-2272af25e327-kube-api-access-rq6kx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-r5grr\" (UID: \"6cdf1158-9e56-4558-b7f5-2272af25e327\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:34.037212 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.037137    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:34.474705 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.474662    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"]
Oct 31 16:50:34.480843 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:50:34.480815    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdf1158_9e56_4558_b7f5_2272af25e327.slice/crio-533cb0b388f0de3b741de8ca29181e77a8103fb48c4949217ae14a5d825b8b23 WatchSource:0}: Error finding container 533cb0b388f0de3b741de8ca29181e77a8103fb48c4949217ae14a5d825b8b23: Status 404 returned error can't find the container with id 533cb0b388f0de3b741de8ca29181e77a8103fb48c4949217ae14a5d825b8b23
Oct 31 16:50:34.483975 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.483935    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.484070 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.484036    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.484116 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.484089    4519 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.985342 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.985304    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" event={"ID":"6cdf1158-9e56-4558-b7f5-2272af25e327","Type":"ContainerStarted","Data":"0cfd835ec10563cd33561a62f0cc5eb9597d0a03f6cc85bf8a2185074d99dbe2"}
Oct 31 16:50:34.985767 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:34.985359    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" event={"ID":"6cdf1158-9e56-4558-b7f5-2272af25e327","Type":"ContainerStarted","Data":"533cb0b388f0de3b741de8ca29181e77a8103fb48c4949217ae14a5d825b8b23"}
Oct 31 16:50:35.037558 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:35.037525    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:35.039049 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:35.039029    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:35.039101 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:35.039076    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:35.192478 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:35.191760    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podStartSLOduration=2.191744565 podStartE2EDuration="2.191744565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:50:35.142163146 +0000 UTC m=+2878.842092460" watchObservedRunningTime="2026-10-31 16:50:35.191744565 +0000 UTC m=+2878.891673876"
Oct 31 16:50:36.037513 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:36.037476    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:36.037933 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:36.037528    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:37.038074 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:37.038034    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:37.038447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:37.038093    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:38.037908 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:38.037868    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:38.038110 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:38.037936    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:39.038165 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:39.038127    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:39.038691 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:39.038185    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:40.038177 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:40.038141    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:40.038603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:40.038193    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:40.811722 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:40.811689    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:50:41.038410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:41.038378    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:41.038855 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:41.038428    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:42.037840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:42.037800    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 16:50:42.038269 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:42.038236    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 16:50:43.247639 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:43.247585    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:43.248180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:43.247651    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:44.039073 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:44.039029    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:44.039279 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:44.039115    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:45.079658 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:45.079612    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:45.080199 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:45.079676    4519 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:45.330703 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:45.330615    4519 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:45.331077 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:45.331049    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:45.331854 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:50:45.331837    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr"
Oct 31 16:50:54.003885 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:50:54.003847    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:51:00.047447 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:00.047411    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"]
Oct 31 16:51:00.094713 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:00.094682    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f89a9b-zb28v"]
Oct 31 16:51:00.438191 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:00.438113    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12" path="/var/lib/kubelet/pods/a7cfb6e6-8fb7-4f99-8cdf-441fc3ac5c12/volumes"
Oct 31 16:51:03.775656 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:03.775622    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:51:13.042410 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:13.042372    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"]
Oct 31 16:51:13.078696 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:13.078662    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1870493f-hdxxh"]
Oct 31 16:51:14.437678 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:14.437639    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72" path="/var/lib/kubelet/pods/058aa6c4-b6e9-418b-a9cb-c9fc8b9e0e72/volumes"
Oct 31 16:51:22.583870 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:22.583792    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:51:24.056339 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:51:24.056303    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:51:54.003895 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:51:54.003852    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:51:54.394200 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:51:54.393799    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:52:05.203134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:05.203097    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:52:17.962326 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:17.962289    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:52:24.003319 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:52:24.003277    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:52:39.467475 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:39.467435    4519 scope.go:122] "RemoveContainer" containerID="89ba04642677a0878f054d918dbd59c254571c3d5160ddb77b1c8c9a65e65016"
Oct 31 16:52:39.476291 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:39.476270    4519 scope.go:122] "RemoveContainer" containerID="f066e33920b98886ec2756f96b5881b56caa47be4f16e165a645c5987d3a955d"
Oct 31 16:52:39.484192 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:39.484170    4519 scope.go:122] "RemoveContainer" containerID="7afb7345e965c308d154d37d236f4ce4509cb7e97f0dac509ce03a2ce0e7ae85"
Oct 31 16:52:40.206989 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:40.206952    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:52:49.767106 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:52:49.767019    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:52:54.003183 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:52:54.003143    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:53:08.297441 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:08.297407    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:53:20.797684 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:20.797648    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:53:24.004165 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:53:24.004127    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:53:25.043686 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:25.043647    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"]
Oct 31 16:53:25.092239 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:25.092205    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8969fb08-qchj2"]
Oct 31 16:53:26.438128 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:26.438096    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88adb615-4145-4517-94de-2d27daa62814" path="/var/lib/kubelet/pods/88adb615-4145-4517-94de-2d27daa62814/volumes"
Oct 31 16:53:39.511030 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:39.510996    4519 scope.go:122] "RemoveContainer" containerID="2c7e50a32b19048985c8f751010aa64fbcc333fae139bae4e9a582602067b893"
Oct 31 16:53:41.006596 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:53:41.006560    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:53:54.004786 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:53:54.004747    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:54:03.105078 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:54:03.105040    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:54:16.279834 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:54:16.279800    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:54:24.055683 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:54:24.055590    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:54:27.366657 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:54:27.366623    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:54:39.531358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:54:39.531329    4519 scope.go:122] "RemoveContainer" containerID="d318e4c9c9d840feaf74ace1bd9fb73ebea5fb84f3a96e793b1d2180e7819b1b"
Oct 31 16:54:39.539938 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:54:39.539916    4519 scope.go:122] "RemoveContainer" containerID="0c3a6dd3c7e09ee3ffbc70c1ea3d638f3b42fdaef1998bdc521b2b1aea21cd96"
Oct 31 16:54:50.188292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:54:50.188253    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:54:54.004028 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:54:54.003989    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:55:00.196594 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:55:00.194251    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:55:14.167858 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:55:14.167818    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:55:24.003768 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:55:24.003727    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:55:35.998111 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:55:35.998068    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:55:46.572603 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:55:46.572564    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:55:54.003368 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:55:54.003318    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:56:10.980905 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:56:10.980868    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:56:21.394311 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:56:21.394270    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:56:24.004032 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:56:24.003996    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:56:48.880957 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:56:48.880913    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:56:54.003374 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:56:54.003325    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:57:06.603942 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:06.603909    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:57:24.003311 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:57:24.003234    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:57:32.516699 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:32.516660    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:57:46.757038 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.757003    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"]
Oct 31 16:57:46.760871 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.760847    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.786876 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.786850    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.787044 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.786885    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745q2\" (UniqueName: \"kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.787122 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.787030    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.811085 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.811059    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:57:46.858134 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.858102    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:57:46.858380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.858118    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:57:46.858380 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.858134    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:57:46.860184 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.860159    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"]
Oct 31 16:57:46.888440 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.888406    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.888620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.888511    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.888620 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.888532    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745q2\" (UniqueName: \"kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.889100 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.889078    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.890879 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.890855    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:46.942180 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:46.942146    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745q2\" (UniqueName: \"kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2\") pod \"maas-api-revoke-sub-1a1045ec-7qzf9\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:47.073954 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:47.073874    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:47.446137 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:47.446048    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"]
Oct 31 16:57:47.452920 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:57:47.452888    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957ef00f_f3da_4473_b1ed_035015be57aa.slice/crio-a4e40362a4ec8bb2d82259a6a2fa81e080940cc2de4da4a34d0a6f687b4766b7 WatchSource:0}: Error finding container a4e40362a4ec8bb2d82259a6a2fa81e080940cc2de4da4a34d0a6f687b4766b7: Status 404 returned error can't find the container with id a4e40362a4ec8bb2d82259a6a2fa81e080940cc2de4da4a34d0a6f687b4766b7
Oct 31 16:57:48.337307 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:48.337269    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9" event={"ID":"957ef00f-f3da-4473-b1ed-035015be57aa","Type":"ContainerStarted","Data":"a7e54526e684eb36c56dc67fa74fc70f316e8c6b56916b3a514e80bb1db78832"}
Oct 31 16:57:48.337721 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:48.337327    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9" event={"ID":"957ef00f-f3da-4473-b1ed-035015be57aa","Type":"ContainerStarted","Data":"a4e40362a4ec8bb2d82259a6a2fa81e080940cc2de4da4a34d0a6f687b4766b7"}
Oct 31 16:57:48.445288 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:48.445241    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9" podStartSLOduration=2.445226876 podStartE2EDuration="2.445226876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:57:48.443893805 +0000 UTC m=+3312.143823116" watchObservedRunningTime="2026-10-31 16:57:48.445226876 +0000 UTC m=+3312.145156186"
Oct 31 16:57:49.344021 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:49.343979    4519 generic.go:356] "Generic (PLEG): container finished" podID="957ef00f-f3da-4473-b1ed-035015be57aa" containerID="a7e54526e684eb36c56dc67fa74fc70f316e8c6b56916b3a514e80bb1db78832" exitCode=0
Oct 31 16:57:49.344489 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:49.344072    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9" event={"ID":"957ef00f-f3da-4473-b1ed-035015be57aa","Type":"ContainerDied","Data":"a7e54526e684eb36c56dc67fa74fc70f316e8c6b56916b3a514e80bb1db78832"}
Oct 31 16:57:50.491948 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.491925    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:50.627743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.627660    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2\" (UniqueName: \"kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2\") pod \"957ef00f-f3da-4473-b1ed-035015be57aa\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") "
Oct 31 16:57:50.627743 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.627703    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca\") pod \"957ef00f-f3da-4473-b1ed-035015be57aa\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") "
Oct 31 16:57:50.628007 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.627794    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca\") pod \"957ef00f-f3da-4473-b1ed-035015be57aa\" (UID: \"957ef00f-f3da-4473-b1ed-035015be57aa\") "
Oct 31 16:57:50.628138 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.628115    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca" pod "957ef00f-f3da-4473-b1ed-035015be57aa" (UID: "957ef00f-f3da-4473-b1ed-035015be57aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:57:50.628215 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.628198    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:57:50.629902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.629878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2" pod "957ef00f-f3da-4473-b1ed-035015be57aa" (UID: "957ef00f-f3da-4473-b1ed-035015be57aa"). InnerVolumeSpecName "kube-api-access-745q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:57:50.629902 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.629873    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca" pod "957ef00f-f3da-4473-b1ed-035015be57aa" (UID: "957ef00f-f3da-4473-b1ed-035015be57aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:57:50.729198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.729163    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745q2\" (UniqueName: \"kubernetes.io/projected/957ef00f-f3da-4473-b1ed-035015be57aa-kube-api-access-745q2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:57:50.729198 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:50.729197    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957ef00f-f3da-4473-b1ed-035015be57aa-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:57:51.354551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:51.354518    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9" event={"ID":"957ef00f-f3da-4473-b1ed-035015be57aa","Type":"ContainerDied","Data":"a4e40362a4ec8bb2d82259a6a2fa81e080940cc2de4da4a34d0a6f687b4766b7"}
Oct 31 16:57:51.354551 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:51.354551    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"
Oct 31 16:57:51.354781 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:51.354563    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e40362a4ec8bb2d82259a6a2fa81e080940cc2de4da4a34d0a6f687b4766b7"
Oct 31 16:57:54.003890 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:57:54.003846    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:57:59.179840 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.179805    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"]
Oct 31 16:57:59.180337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.180244    4519 cpu_manager.go:414] "containerMap: removing container" podUID="957ef00f-f3da-4473-b1ed-035015be57aa" containerName="revoke-keys"
Oct 31 16:57:59.180337 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.180258    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957ef00f-f3da-4473-b1ed-035015be57aa" containerName="revoke-keys"
Oct 31 16:57:59.180423 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.180339    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="957ef00f-f3da-4473-b1ed-035015be57aa" containerName="revoke-keys"
Oct 31 16:57:59.183889 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.183863    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.233939 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.233902    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:57:59.258896 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.258869    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:57:59.280914 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.280888    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:57:59.281066 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.280921    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:57:59.283183 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.283156    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"]
Oct 31 16:57:59.313633 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.313596    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.313797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.313663    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.313797 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.313743    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqjr\" (UniqueName: \"kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.415006 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.414970    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.415188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.415032    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.415188 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.415055    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqjr\" (UniqueName: \"kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.415805 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.415786    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.417292 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.417267    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.468575 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.468547    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqjr\" (UniqueName: \"kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.497204 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.497174    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:57:59.864539 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.864428    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"]
Oct 31 16:57:59.871059 ip-10-0-129-180 kubenswrapper[4519]: W1031 16:57:59.871014    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69adfc5e_3828_404f_98c3_25f9c737d5ce.slice/crio-39e881a710e962300efcb262dcf79a270270af20ea48f416bb7ff556ba1c4751 WatchSource:0}: Error finding container 39e881a710e962300efcb262dcf79a270270af20ea48f416bb7ff556ba1c4751: Status 404 returned error can't find the container with id 39e881a710e962300efcb262dcf79a270270af20ea48f416bb7ff556ba1c4751
Oct 31 16:57:59.904144 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:57:59.904117    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:58:00.397733 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:00.397695    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5" event={"ID":"69adfc5e-3828-404f-98c3-25f9c737d5ce","Type":"ContainerStarted","Data":"c03762880eef748ecf01b559aefd001ff792a20eb868afba4032567c1a28f59e"}
Oct 31 16:58:00.398149 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:00.397756    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5" event={"ID":"69adfc5e-3828-404f-98c3-25f9c737d5ce","Type":"ContainerStarted","Data":"39e881a710e962300efcb262dcf79a270270af20ea48f416bb7ff556ba1c4751"}
Oct 31 16:58:00.504973 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:00.504926    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5" podStartSLOduration=1.504913342 podStartE2EDuration="1.504913342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:58:00.504270769 +0000 UTC m=+3324.204200083" watchObservedRunningTime="2026-10-31 16:58:00.504913342 +0000 UTC m=+3324.204842652"
Oct 31 16:58:01.403780 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:01.403741    4519 generic.go:356] "Generic (PLEG): container finished" podID="69adfc5e-3828-404f-98c3-25f9c737d5ce" containerID="c03762880eef748ecf01b559aefd001ff792a20eb868afba4032567c1a28f59e" exitCode=0
Oct 31 16:58:01.404276 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:01.403828    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5" event={"ID":"69adfc5e-3828-404f-98c3-25f9c737d5ce","Type":"ContainerDied","Data":"c03762880eef748ecf01b559aefd001ff792a20eb868afba4032567c1a28f59e"}
Oct 31 16:58:02.547060 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.547029    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:58:02.647229 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.647186    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca\") pod \"69adfc5e-3828-404f-98c3-25f9c737d5ce\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") "
Oct 31 16:58:02.647438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.647257    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr\" (UniqueName: \"kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr\") pod \"69adfc5e-3828-404f-98c3-25f9c737d5ce\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") "
Oct 31 16:58:02.647438 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.647296    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca\") pod \"69adfc5e-3828-404f-98c3-25f9c737d5ce\" (UID: \"69adfc5e-3828-404f-98c3-25f9c737d5ce\") "
Oct 31 16:58:02.647717 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.647694    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca" pod "69adfc5e-3828-404f-98c3-25f9c737d5ce" (UID: "69adfc5e-3828-404f-98c3-25f9c737d5ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:58:02.649358 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.649335    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr" pod "69adfc5e-3828-404f-98c3-25f9c737d5ce" (UID: "69adfc5e-3828-404f-98c3-25f9c737d5ce"). InnerVolumeSpecName "kube-api-access-frqjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:02.649482 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.649341    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca" pod "69adfc5e-3828-404f-98c3-25f9c737d5ce" (UID: "69adfc5e-3828-404f-98c3-25f9c737d5ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:58:02.749185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.749151    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqjr\" (UniqueName: \"kubernetes.io/projected/69adfc5e-3828-404f-98c3-25f9c737d5ce-kube-api-access-frqjr\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:02.749185 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.749182    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:02.749401 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:02.749196    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69adfc5e-3828-404f-98c3-25f9c737d5ce-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:03.414583 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:03.414547    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5" event={"ID":"69adfc5e-3828-404f-98c3-25f9c737d5ce","Type":"ContainerDied","Data":"39e881a710e962300efcb262dcf79a270270af20ea48f416bb7ff556ba1c4751"}
Oct 31 16:58:03.414748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:03.414595    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e881a710e962300efcb262dcf79a270270af20ea48f416bb7ff556ba1c4751"
Oct 31 16:58:03.414748 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:03.414591    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"
Oct 31 16:58:17.095814 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:17.095781    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"]
Oct 31 16:58:17.142516 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:17.142481    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f191a7-q4gh5"]
Oct 31 16:58:18.437881 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:18.437837    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69adfc5e-3828-404f-98c3-25f9c737d5ce" path="/var/lib/kubelet/pods/69adfc5e-3828-404f-98c3-25f9c737d5ce/volumes"
Oct 31 16:58:19.501088 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:19.501053    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:58:24.003939 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:58:24.003894    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:58:39.869230 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:39.869189    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:58:50.478333 ip-10-0-129-180 kubenswrapper[4519]: I1031 16:58:50.478252    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 16:58:54.004634 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:58:54.004590    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:59:24.005675 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:59:24.005633    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 16:59:54.002705 ip-10-0-129-180 kubenswrapper[4519]: E1031 16:59:54.002668    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:00:24.003510 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:00:24.003414    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:00:24.969907 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:00:24.969873    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:00:41.986051 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:00:41.986016    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:00:51.004394 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:00:51.004356    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:00:54.006966 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:00:54.006924    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:01:11.102740 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:01:11.102703    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:01:24.003480 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:01:24.003418    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:01:30.902351 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:01:30.902316    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:01:49.489746 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:01:49.489662    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:01:54.003005 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:01:54.002970    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:02:15.265863 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:15.265822    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:02:24.004286 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:02:24.004249    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:02:29.901062 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:29.901025    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"]
Oct 31 17:02:29.901566 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:29.901548    4519 cpu_manager.go:414] "containerMap: removing container" podUID="69adfc5e-3828-404f-98c3-25f9c737d5ce" containerName="revoke-keys"
Oct 31 17:02:29.901621 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:29.901568    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69adfc5e-3828-404f-98c3-25f9c737d5ce" containerName="revoke-keys"
Oct 31 17:02:29.901659 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:29.901635    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="69adfc5e-3828-404f-98c3-25f9c737d5ce" containerName="revoke-keys"
Oct 31 17:02:29.905203 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:29.905183    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:29.957061 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:29.957027    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"]
Oct 31 17:02:30.001727 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.001696    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:02:30.001727 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.001697    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:02:30.001956 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.001783    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:02:30.002088 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.002071    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:02:30.071874 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.071842    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.072044 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.071901    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.072044 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.072024    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcr4\" (UniqueName: \"kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.173247 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.173157    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.173432 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.173323    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcr4\" (UniqueName: \"kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.173432 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.173396    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.174015 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.173994    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.175635 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.175614    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.227649 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.227622    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcr4\" (UniqueName: \"kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4\") pod \"maas-api-revoke-sub-0f0929de-vb2cg\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.517772 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.517739    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:30.885310 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:30.885234    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"]
Oct 31 17:02:30.891864 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:02:30.891826    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40be721_c04f_423f_aed7_7356efb755a7.slice/crio-b64950b1e08fb29207efdc1cae66cd94576b43ac1b3fb345c55dd25b6cc68e7c WatchSource:0}: Error finding container b64950b1e08fb29207efdc1cae66cd94576b43ac1b3fb345c55dd25b6cc68e7c: Status 404 returned error can't find the container with id b64950b1e08fb29207efdc1cae66cd94576b43ac1b3fb345c55dd25b6cc68e7c
Oct 31 17:02:31.775385 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:31.775339    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg" event={"ID":"e40be721-c04f-423f-aed7-7356efb755a7","Type":"ContainerStarted","Data":"2c58615073bab253b0e6aac2a6b5cef5b4f87c4847891de5c7d080183ce2b455"}
Oct 31 17:02:31.775826 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:31.775407    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg" event={"ID":"e40be721-c04f-423f-aed7-7356efb755a7","Type":"ContainerStarted","Data":"b64950b1e08fb29207efdc1cae66cd94576b43ac1b3fb345c55dd25b6cc68e7c"}
Oct 31 17:02:31.884792 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:31.884737    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg" podStartSLOduration=2.8847211230000003 podStartE2EDuration="2.884721123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:02:31.883807721 +0000 UTC m=+3595.583737032" watchObservedRunningTime="2026-10-31 17:02:31.884721123 +0000 UTC m=+3595.584650434"
Oct 31 17:02:32.781971 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:32.781938    4519 generic.go:356] "Generic (PLEG): container finished" podID="e40be721-c04f-423f-aed7-7356efb755a7" containerID="2c58615073bab253b0e6aac2a6b5cef5b4f87c4847891de5c7d080183ce2b455" exitCode=0
Oct 31 17:02:32.782382 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:32.782020    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg" event={"ID":"e40be721-c04f-423f-aed7-7356efb755a7","Type":"ContainerDied","Data":"2c58615073bab253b0e6aac2a6b5cef5b4f87c4847891de5c7d080183ce2b455"}
Oct 31 17:02:33.921310 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:33.921285    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:34.009817 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.009782    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca\") pod \"e40be721-c04f-423f-aed7-7356efb755a7\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") "
Oct 31 17:02:34.009981 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.009838    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca\") pod \"e40be721-c04f-423f-aed7-7356efb755a7\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") "
Oct 31 17:02:34.009981 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.009871    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4\" (UniqueName: \"kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4\") pod \"e40be721-c04f-423f-aed7-7356efb755a7\" (UID: \"e40be721-c04f-423f-aed7-7356efb755a7\") "
Oct 31 17:02:34.010253 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.010223    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca" pod "e40be721-c04f-423f-aed7-7356efb755a7" (UID: "e40be721-c04f-423f-aed7-7356efb755a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:02:34.011902 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.011878    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca" pod "e40be721-c04f-423f-aed7-7356efb755a7" (UID: "e40be721-c04f-423f-aed7-7356efb755a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:02:34.012019 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.012008    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4" pod "e40be721-c04f-423f-aed7-7356efb755a7" (UID: "e40be721-c04f-423f-aed7-7356efb755a7"). InnerVolumeSpecName "kube-api-access-zwcr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:02:34.111014 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.110931    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:34.111014 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.110961    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e40be721-c04f-423f-aed7-7356efb755a7-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:34.111014 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.110973    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcr4\" (UniqueName: \"kubernetes.io/projected/e40be721-c04f-423f-aed7-7356efb755a7-kube-api-access-zwcr4\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:34.792606 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.792576    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"
Oct 31 17:02:34.792898 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.792615    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg" event={"ID":"e40be721-c04f-423f-aed7-7356efb755a7","Type":"ContainerDied","Data":"b64950b1e08fb29207efdc1cae66cd94576b43ac1b3fb345c55dd25b6cc68e7c"}
Oct 31 17:02:34.792898 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:34.792674    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b64950b1e08fb29207efdc1cae66cd94576b43ac1b3fb345c55dd25b6cc68e7c"
Oct 31 17:02:41.625150 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.625115    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"]
Oct 31 17:02:41.625615 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.625600    4519 cpu_manager.go:414] "containerMap: removing container" podUID="e40be721-c04f-423f-aed7-7356efb755a7" containerName="revoke-keys"
Oct 31 17:02:41.625615 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.625616    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40be721-c04f-423f-aed7-7356efb755a7" containerName="revoke-keys"
Oct 31 17:02:41.625707 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.625686    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40be721-c04f-423f-aed7-7356efb755a7" containerName="revoke-keys"
Oct 31 17:02:41.628796 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.628778    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.680858 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.680827    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"]
Oct 31 17:02:41.687003 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.686968    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgws2\" (UniqueName: \"kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.687172 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.687131    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.687257 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.687178    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.726200 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.726165    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:02:41.726200 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.726169    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:02:41.726415 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.726212    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:02:41.726415 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.726242    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:02:41.788569 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.788531    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.788569 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.788575    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.788800 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.788631    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgws2\" (UniqueName: \"kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.789264 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.789232    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.790856 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.790832    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.853263 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.853225    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgws2\" (UniqueName: \"kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:41.941060 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:41.940978    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:42.311529 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:42.311495    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"]
Oct 31 17:02:42.318031 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:02:42.318002    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac259f3_c6cb_4703_8475_0d262732d907.slice/crio-8135a0097f074a5d4b917db8d4156f1e92ea64f39c4c3c83c9927ed364b27fd9 WatchSource:0}: Error finding container 8135a0097f074a5d4b917db8d4156f1e92ea64f39c4c3c83c9927ed364b27fd9: Status 404 returned error can't find the container with id 8135a0097f074a5d4b917db8d4156f1e92ea64f39c4c3c83c9927ed364b27fd9
Oct 31 17:02:42.790434 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:42.790393    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:02:42.832100 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:42.832013    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2" event={"ID":"bac259f3-c6cb-4703-8475-0d262732d907","Type":"ContainerStarted","Data":"5d5a05d36baea4980a48c4ba0a37151e0ed63f06f7b690c10504f9b94e45aa45"}
Oct 31 17:02:42.832100 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:42.832070    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2" event={"ID":"bac259f3-c6cb-4703-8475-0d262732d907","Type":"ContainerStarted","Data":"8135a0097f074a5d4b917db8d4156f1e92ea64f39c4c3c83c9927ed364b27fd9"}
Oct 31 17:02:42.940592 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:42.940534    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2" podStartSLOduration=1.940521062 podStartE2EDuration="1.940521062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:02:42.939333019 +0000 UTC m=+3606.639262331" watchObservedRunningTime="2026-10-31 17:02:42.940521062 +0000 UTC m=+3606.640450373"
Oct 31 17:02:43.837948 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:43.837916    4519 generic.go:356] "Generic (PLEG): container finished" podID="bac259f3-c6cb-4703-8475-0d262732d907" containerID="5d5a05d36baea4980a48c4ba0a37151e0ed63f06f7b690c10504f9b94e45aa45" exitCode=0
Oct 31 17:02:43.838363 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:43.837989    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2" event={"ID":"bac259f3-c6cb-4703-8475-0d262732d907","Type":"ContainerDied","Data":"5d5a05d36baea4980a48c4ba0a37151e0ed63f06f7b690c10504f9b94e45aa45"}
Oct 31 17:02:44.983290 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:44.983260    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:45.022759 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.022727    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca\") pod \"bac259f3-c6cb-4703-8475-0d262732d907\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") "
Oct 31 17:02:45.022927 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.022803    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca\") pod \"bac259f3-c6cb-4703-8475-0d262732d907\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") "
Oct 31 17:02:45.022927 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.022905    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2\" (UniqueName: \"kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2\") pod \"bac259f3-c6cb-4703-8475-0d262732d907\" (UID: \"bac259f3-c6cb-4703-8475-0d262732d907\") "
Oct 31 17:02:45.023169 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.023145    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca" pod "bac259f3-c6cb-4703-8475-0d262732d907" (UID: "bac259f3-c6cb-4703-8475-0d262732d907"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:02:45.024899 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.024871    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca" pod "bac259f3-c6cb-4703-8475-0d262732d907" (UID: "bac259f3-c6cb-4703-8475-0d262732d907"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:02:45.025028 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.025012    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2" pod "bac259f3-c6cb-4703-8475-0d262732d907" (UID: "bac259f3-c6cb-4703-8475-0d262732d907"). InnerVolumeSpecName "kube-api-access-rgws2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:02:45.123993 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.123913    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgws2\" (UniqueName: \"kubernetes.io/projected/bac259f3-c6cb-4703-8475-0d262732d907-kube-api-access-rgws2\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:45.123993 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.123942    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:45.123993 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.123952    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac259f3-c6cb-4703-8475-0d262732d907-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:45.851073 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.851044    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"
Oct 31 17:02:45.851256 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.851045    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2" event={"ID":"bac259f3-c6cb-4703-8475-0d262732d907","Type":"ContainerDied","Data":"8135a0097f074a5d4b917db8d4156f1e92ea64f39c4c3c83c9927ed364b27fd9"}
Oct 31 17:02:45.851256 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:45.851163    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8135a0097f074a5d4b917db8d4156f1e92ea64f39c4c3c83c9927ed364b27fd9"
Oct 31 17:02:51.048465 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:51.048425    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"]
Oct 31 17:02:51.095618 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:51.095586    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1045ec-7qzf9"]
Oct 31 17:02:52.437718 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:52.437683    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957ef00f-f3da-4473-b1ed-035015be57aa" path="/var/lib/kubelet/pods/957ef00f-f3da-4473-b1ed-035015be57aa/volumes"
Oct 31 17:02:54.002426 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:02:54.002390    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:02:59.524874 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:59.524835    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"]
Oct 31 17:02:59.572318 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:59.572271    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a7bea2-9j5h2"]
Oct 31 17:02:59.967705 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:02:59.967673    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:03:00.438447 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:03:00.438354    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac259f3-c6cb-4703-8475-0d262732d907" path="/var/lib/kubelet/pods/bac259f3-c6cb-4703-8475-0d262732d907/volumes"
Oct 31 17:03:17.891920 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:03:17.891838    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:03:24.003812 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:03:24.003779    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:03:54.004905 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:03:54.004862    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:03:58.288335 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:03:58.288298    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:04:01.385479 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:01.384550    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-r5grr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:04:01.385479 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:01.384631    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-r5grr" podUID="6cdf1158-9e56-4558-b7f5-2272af25e327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:04:24.004185 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:04:24.004141    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:04:25.707487 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:25.707442    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:04:28.249636 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:28.249592    4519 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5tkrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:04:28.250059 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:28.249674    4519 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5tkrq" podUID="f24f8e20-1030-4e69-96c6-a4a20a579947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:04:39.627162 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:39.627132    4519 scope.go:122] "RemoveContainer" containerID="c03762880eef748ecf01b559aefd001ff792a20eb868afba4032567c1a28f59e"
Oct 31 17:04:39.636232 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:39.636211    4519 scope.go:122] "RemoveContainer" containerID="a7e54526e684eb36c56dc67fa74fc70f316e8c6b56916b3a514e80bb1db78832"
Oct 31 17:04:41.064743 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:41.064709    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:04:52.510010 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:04:52.509930    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:04:54.003668 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:04:54.003626    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:05:00.605962 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:05:00.605930    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:05:10.075037 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:05:10.074999    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:05:24.003319 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:05:24.003277    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:05:31.799294 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:05:31.799257    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:05:54.003781 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:05:54.003733    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:05:56.907900 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:05:56.907856    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:06:19.416851 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.416769    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"]
Oct 31 17:06:19.417338 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.417191    4519 cpu_manager.go:414] "containerMap: removing container" podUID="bac259f3-c6cb-4703-8475-0d262732d907" containerName="revoke-keys"
Oct 31 17:06:19.417338 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.417204    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac259f3-c6cb-4703-8475-0d262732d907" containerName="revoke-keys"
Oct 31 17:06:19.417338 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.417291    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac259f3-c6cb-4703-8475-0d262732d907" containerName="revoke-keys"
Oct 31 17:06:19.420525 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.420506    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.474154 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.474121    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"]
Oct 31 17:06:19.520684 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.520650    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:06:19.520684 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.520655    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:06:19.520684 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.520664    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:06:19.520947 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.520746    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tdnpj\""
Oct 31 17:06:19.539056 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.539026    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6k8\" (UniqueName: \"kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.539236 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.539146    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.539236 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.539210    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.640193 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.640160    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.640380 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.640207    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.640380 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.640247    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6k8\" (UniqueName: \"kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.640851 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.640817    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.642593 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.642571    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.695094 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.695018    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6k8\" (UniqueName: \"kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8\") pod \"maas-api-revoke-keys-e2e-ait-c6ac3112-q829x\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:19.733577 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:19.733544    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:20.100399 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:20.100364    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"]
Oct 31 17:06:20.106422 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:06:20.106388    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd259da85_32b6_4e41_8f15_c032adc2e715.slice/crio-5f44c3c9ba90da5f1cc3a607f8f7854b5b76ce7c0a53726c7839aa721a154a2b WatchSource:0}: Error finding container 5f44c3c9ba90da5f1cc3a607f8f7854b5b76ce7c0a53726c7839aa721a154a2b: Status 404 returned error can't find the container with id 5f44c3c9ba90da5f1cc3a607f8f7854b5b76ce7c0a53726c7839aa721a154a2b
Oct 31 17:06:21.106499 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:21.106446    4519 generic.go:356] "Generic (PLEG): container finished" podID="d259da85-32b6-4e41-8f15-c032adc2e715" containerID="bd1eb0df3efa8859325bdedbe7b6ec579e923c5e3103b5c1c4a18112b24f0b2c" exitCode=0
Oct 31 17:06:21.106960 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:21.106534    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x" event={"ID":"d259da85-32b6-4e41-8f15-c032adc2e715","Type":"ContainerDied","Data":"bd1eb0df3efa8859325bdedbe7b6ec579e923c5e3103b5c1c4a18112b24f0b2c"}
Oct 31 17:06:21.106960 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:21.106588    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x" event={"ID":"d259da85-32b6-4e41-8f15-c032adc2e715","Type":"ContainerStarted","Data":"5f44c3c9ba90da5f1cc3a607f8f7854b5b76ce7c0a53726c7839aa721a154a2b"}
Oct 31 17:06:22.255099 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.255072    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:22.267152 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.267129    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8\" (UniqueName: \"kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8\") pod \"d259da85-32b6-4e41-8f15-c032adc2e715\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") "
Oct 31 17:06:22.267237 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.267224    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca\") pod \"d259da85-32b6-4e41-8f15-c032adc2e715\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") "
Oct 31 17:06:22.267300 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.267286    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca\") pod \"d259da85-32b6-4e41-8f15-c032adc2e715\" (UID: \"d259da85-32b6-4e41-8f15-c032adc2e715\") "
Oct 31 17:06:22.267604 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.267579    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca" pod "d259da85-32b6-4e41-8f15-c032adc2e715" (UID: "d259da85-32b6-4e41-8f15-c032adc2e715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:06:22.269311 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.269291    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca" pod "d259da85-32b6-4e41-8f15-c032adc2e715" (UID: "d259da85-32b6-4e41-8f15-c032adc2e715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:06:22.269383 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.269355    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8" pod "d259da85-32b6-4e41-8f15-c032adc2e715" (UID: "d259da85-32b6-4e41-8f15-c032adc2e715"). InnerVolumeSpecName "kube-api-access-xm6k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:06:22.368957 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.368883    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:22.368957 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.368913    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6k8\" (UniqueName: \"kubernetes.io/projected/d259da85-32b6-4e41-8f15-c032adc2e715-kube-api-access-xm6k8\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:22.368957 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:22.368924    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d259da85-32b6-4e41-8f15-c032adc2e715-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:23.118487 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:23.118440    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"
Oct 31 17:06:23.118487 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:23.118477    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x" event={"ID":"d259da85-32b6-4e41-8f15-c032adc2e715","Type":"ContainerDied","Data":"5f44c3c9ba90da5f1cc3a607f8f7854b5b76ce7c0a53726c7839aa721a154a2b"}
Oct 31 17:06:23.118758 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:23.118529    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f44c3c9ba90da5f1cc3a607f8f7854b5b76ce7c0a53726c7839aa721a154a2b"
Oct 31 17:06:24.003985 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:06:24.003945    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:06:35.739964 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:35.739931    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"]
Oct 31 17:06:35.798187 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:35.798136    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:06:35.800164 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:35.800141    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c6ac3112-q829x"]
Oct 31 17:06:36.439334 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:36.439299    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d259da85-32b6-4e41-8f15-c032adc2e715" path="/var/lib/kubelet/pods/d259da85-32b6-4e41-8f15-c032adc2e715/volumes"
Oct 31 17:06:54.004349 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:06:54.004311    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:06:59.007743 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:06:59.007702    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:07:24.004194 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:07:24.004149    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:07:34.042070 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:07:34.042034    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"]
Oct 31 17:07:34.089205 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:07:34.089170    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f0929de-vb2cg"]
Oct 31 17:07:34.437371 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:07:34.437293    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40be721-c04f-423f-aed7-7356efb755a7" path="/var/lib/kubelet/pods/e40be721-c04f-423f-aed7-7356efb755a7/volumes"
Oct 31 17:07:54.003890 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:07:54.003850    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:08:24.003531 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:08:24.003493    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:08:30.894028 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.893992    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"]
Oct 31 17:08:30.894444 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.894417    4519 cpu_manager.go:414] "containerMap: removing container" podUID="d259da85-32b6-4e41-8f15-c032adc2e715" containerName="revoke-keys"
Oct 31 17:08:30.894444 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.894429    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d259da85-32b6-4e41-8f15-c032adc2e715" containerName="revoke-keys"
Oct 31 17:08:30.894549 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.894537    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="d259da85-32b6-4e41-8f15-c032adc2e715" containerName="revoke-keys"
Oct 31 17:08:30.897674 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.897656    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:30.949060 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.949030    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"]
Oct 31 17:08:30.993945 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.993912    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:08:30.993945 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.993932    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:08:30.994152 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.993954    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:08:30.994589 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:30.994568    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tdnpj\""
Oct 31 17:08:31.062828 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.062793    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpmk\" (UniqueName: \"kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.063023 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.062836    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.063023 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.062920    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.163728 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.163618    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpmk\" (UniqueName: \"kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.163728 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.163668    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.163728 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.163708    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.165231 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.165197    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.177997 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.177885    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.217480 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.217433    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpmk\" (UniqueName: \"kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk\") pod \"maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.510523 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.510486    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:31.877270 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:31.877178    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"]
Oct 31 17:08:31.883272 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:08:31.883233    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d0fbdb_6a7f_411e_9a4f_7c332c3544e5.slice/crio-289a32dd2c5da27808d312532a8131e8b995a17025a0b901f7fd39423ff7401d WatchSource:0}: Error finding container 289a32dd2c5da27808d312532a8131e8b995a17025a0b901f7fd39423ff7401d: Status 404 returned error can't find the container with id 289a32dd2c5da27808d312532a8131e8b995a17025a0b901f7fd39423ff7401d
Oct 31 17:08:32.783915 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:32.783883    4519 generic.go:356] "Generic (PLEG): container finished" podID="17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" containerID="18d1ff1c3dfe651384cff00b5a8611b4b63a331d6cf151ad1607aced8c49d241" exitCode=0
Oct 31 17:08:32.784285 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:32.783922    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t" event={"ID":"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5","Type":"ContainerDied","Data":"18d1ff1c3dfe651384cff00b5a8611b4b63a331d6cf151ad1607aced8c49d241"}
Oct 31 17:08:32.784285 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:32.783966    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t" event={"ID":"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5","Type":"ContainerStarted","Data":"289a32dd2c5da27808d312532a8131e8b995a17025a0b901f7fd39423ff7401d"}
Oct 31 17:08:33.924979 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:33.924949    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:34.093372 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.093289    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk\" (UniqueName: \"kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk\") pod \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") "
Oct 31 17:08:34.093570 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.093381    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca\") pod \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") "
Oct 31 17:08:34.093570 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.093503    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca\") pod \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\" (UID: \"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5\") "
Oct 31 17:08:34.093709 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.093684    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca" pod "17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" (UID: "17d0fbdb-6a7f-411e-9a4f-7c332c3544e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:08:34.093889 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.093873    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:34.095469 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.095426    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk" pod "17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" (UID: "17d0fbdb-6a7f-411e-9a4f-7c332c3544e5"). InnerVolumeSpecName "kube-api-access-bqpmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:08:34.095572 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.095475    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca" pod "17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" (UID: "17d0fbdb-6a7f-411e-9a4f-7c332c3544e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:08:34.194947 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.194911    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqpmk\" (UniqueName: \"kubernetes.io/projected/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-kube-api-access-bqpmk\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:34.194947 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.194942    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:34.794706 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.794678    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"
Oct 31 17:08:34.794880 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.794674    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t" event={"ID":"17d0fbdb-6a7f-411e-9a4f-7c332c3544e5","Type":"ContainerDied","Data":"289a32dd2c5da27808d312532a8131e8b995a17025a0b901f7fd39423ff7401d"}
Oct 31 17:08:34.794880 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:34.794801    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="289a32dd2c5da27808d312532a8131e8b995a17025a0b901f7fd39423ff7401d"
Oct 31 17:08:39.682993 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:39.682962    4519 scope.go:122] "RemoveContainer" containerID="2c58615073bab253b0e6aac2a6b5cef5b4f87c4847891de5c7d080183ce2b455"
Oct 31 17:08:54.004334 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:08:54.004295    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:08:54.961438 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:54.961403    4519 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"]
Oct 31 17:08:55.008725 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:55.008693    4519 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-713882-sfl7t"]
Oct 31 17:08:56.437893 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:56.437862    4519 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" path="/var/lib/kubelet/pods/17d0fbdb-6a7f-411e-9a4f-7c332c3544e5/volumes"
Oct 31 17:08:57.961045 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:08:57.961013    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:09:08.988127 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:08.988085    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:09:20.198717 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:20.198632    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:09:24.003098 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:09:24.003054    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:09:28.408279 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.408240    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"]
Oct 31 17:09:28.408798 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.408694    4519 cpu_manager.go:414] "containerMap: removing container" podUID="17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" containerName="revoke-keys"
Oct 31 17:09:28.408798 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.408707    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" containerName="revoke-keys"
Oct 31 17:09:28.408798 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.408774    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d0fbdb-6a7f-411e-9a4f-7c332c3544e5" containerName="revoke-keys"
Oct 31 17:09:28.411945 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.411926    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.463996 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.463968    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"]
Oct 31 17:09:28.489609 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.489573    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.489777 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.489688    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.489777 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.489756    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljhm\" (UniqueName: \"kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.509148 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.509124    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:09:28.509148 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.509141    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:09:28.509148 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.509152    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tdnpj\""
Oct 31 17:09:28.509352 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.509211    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:09:28.591253 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.591225    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.591427 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.591263    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.591427 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.591288    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljhm\" (UniqueName: \"kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.591969 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.591945    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.593584 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.593562    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.645529 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.645505    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljhm\" (UniqueName: \"kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm\") pod \"maas-api-revoke-sub-f03629db-tklqt\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:28.724941 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:28.724913    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:29.094848 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:29.094770    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"]
Oct 31 17:09:29.101092 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:09:29.101066    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43892db6_3779_41e1_a334_3093ca248a9b.slice/crio-d714c4df3582e49a1c98aa3a1710322adb4dfd9b8402095293900b17d3628d26 WatchSource:0}: Error finding container d714c4df3582e49a1c98aa3a1710322adb4dfd9b8402095293900b17d3628d26: Status 404 returned error can't find the container with id d714c4df3582e49a1c98aa3a1710322adb4dfd9b8402095293900b17d3628d26
Oct 31 17:09:30.092128 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:30.092091    4519 generic.go:356] "Generic (PLEG): container finished" podID="43892db6-3779-41e1-a334-3093ca248a9b" containerID="38c72fa1867af0eef7c83847c4a4c345e03697613bf93b1ff3651be59d16c900" exitCode=0
Oct 31 17:09:30.092631 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:30.092180    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt" event={"ID":"43892db6-3779-41e1-a334-3093ca248a9b","Type":"ContainerDied","Data":"38c72fa1867af0eef7c83847c4a4c345e03697613bf93b1ff3651be59d16c900"}
Oct 31 17:09:30.092631 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:30.092237    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt" event={"ID":"43892db6-3779-41e1-a334-3093ca248a9b","Type":"ContainerStarted","Data":"d714c4df3582e49a1c98aa3a1710322adb4dfd9b8402095293900b17d3628d26"}
Oct 31 17:09:31.251808 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.251778    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:31.313739 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.313708    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm\" (UniqueName: \"kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm\") pod \"43892db6-3779-41e1-a334-3093ca248a9b\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") "
Oct 31 17:09:31.313950 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.313855    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca\") pod \"43892db6-3779-41e1-a334-3093ca248a9b\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") "
Oct 31 17:09:31.313950 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.313893    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca\") pod \"43892db6-3779-41e1-a334-3093ca248a9b\" (UID: \"43892db6-3779-41e1-a334-3093ca248a9b\") "
Oct 31 17:09:31.314354 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.314321    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca" pod "43892db6-3779-41e1-a334-3093ca248a9b" (UID: "43892db6-3779-41e1-a334-3093ca248a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:09:31.315957 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.315933    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca" pod "43892db6-3779-41e1-a334-3093ca248a9b" (UID: "43892db6-3779-41e1-a334-3093ca248a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:09:31.316678 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.316653    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm" pod "43892db6-3779-41e1-a334-3093ca248a9b" (UID: "43892db6-3779-41e1-a334-3093ca248a9b"). InnerVolumeSpecName "kube-api-access-wljhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:09:31.414922 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.414845    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljhm\" (UniqueName: \"kubernetes.io/projected/43892db6-3779-41e1-a334-3093ca248a9b-kube-api-access-wljhm\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:31.414922 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.414873    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:31.414922 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:31.414886    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43892db6-3779-41e1-a334-3093ca248a9b-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:32.104048 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:32.104016    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt"
Oct 31 17:09:32.104048 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:32.104034    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-tklqt" event={"ID":"43892db6-3779-41e1-a334-3093ca248a9b","Type":"ContainerDied","Data":"d714c4df3582e49a1c98aa3a1710322adb4dfd9b8402095293900b17d3628d26"}
Oct 31 17:09:32.104336 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:32.104112    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d714c4df3582e49a1c98aa3a1710322adb4dfd9b8402095293900b17d3628d26"
Oct 31 17:09:39.703253 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:39.703218    4519 scope.go:122] "RemoveContainer" containerID="5d5a05d36baea4980a48c4ba0a37151e0ed63f06f7b690c10504f9b94e45aa45"
Oct 31 17:09:41.167249 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:41.167217    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:09:49.097081 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.097036    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"]
Oct 31 17:09:49.097703 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.097682    4519 cpu_manager.go:414] "containerMap: removing container" podUID="43892db6-3779-41e1-a334-3093ca248a9b" containerName="revoke-keys"
Oct 31 17:09:49.097800 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.097704    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43892db6-3779-41e1-a334-3093ca248a9b" containerName="revoke-keys"
Oct 31 17:09:49.097862 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.097853    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="43892db6-3779-41e1-a334-3093ca248a9b" containerName="revoke-keys"
Oct 31 17:09:49.101065 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.101043    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.153595 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.153561    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"]
Oct 31 17:09:49.198267 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.198231    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:09:49.198267 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.198251    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:09:49.198267 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.198263    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tdnpj\""
Oct 31 17:09:49.198535 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.198274    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:09:49.232004 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.231978    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.232143 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.232092    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfljb\" (UniqueName: \"kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.232183 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.232152    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.333418 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.333388    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfljb\" (UniqueName: \"kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.333587 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.333438    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.333587 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.333497    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.334140 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.334115    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.335741 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.335717    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.386806 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.386748    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfljb\" (UniqueName: \"kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb\") pod \"maas-api-revoke-sub-0165d7d2-ngglb\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.415170 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.415147    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:49.782018 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:49.781986    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"]
Oct 31 17:09:49.788050 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:09:49.788023    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350406cf_db3a_491d_8a0d_79b0af35b6cd.slice/crio-3193ab62df410be8bd8b3b858136f4e6560cbb5bd6ec7fe20ff4b31b536185e8 WatchSource:0}: Error finding container 3193ab62df410be8bd8b3b858136f4e6560cbb5bd6ec7fe20ff4b31b536185e8: Status 404 returned error can't find the container with id 3193ab62df410be8bd8b3b858136f4e6560cbb5bd6ec7fe20ff4b31b536185e8
Oct 31 17:09:50.219877 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:50.219842    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb" event={"ID":"350406cf-db3a-491d-8a0d-79b0af35b6cd","Type":"ContainerStarted","Data":"81f54c4fd5eb6782451c6f238851495eae7e47236e8936956d6c6f40907500dd"}
Oct 31 17:09:50.220278 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:50.219937    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb" event={"ID":"350406cf-db3a-491d-8a0d-79b0af35b6cd","Type":"ContainerStarted","Data":"3193ab62df410be8bd8b3b858136f4e6560cbb5bd6ec7fe20ff4b31b536185e8"}
Oct 31 17:09:50.326327 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:50.326274    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb" podStartSLOduration=1.326261591 podStartE2EDuration="1.326261591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:09:50.325828014 +0000 UTC m=+4034.025757324" watchObservedRunningTime="2026-10-31 17:09:50.326261591 +0000 UTC m=+4034.026190901"
Oct 31 17:09:51.226080 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:51.226041    4519 generic.go:356] "Generic (PLEG): container finished" podID="350406cf-db3a-491d-8a0d-79b0af35b6cd" containerID="81f54c4fd5eb6782451c6f238851495eae7e47236e8936956d6c6f40907500dd" exitCode=0
Oct 31 17:09:51.226508 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:51.226094    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb" event={"ID":"350406cf-db3a-491d-8a0d-79b0af35b6cd","Type":"ContainerDied","Data":"81f54c4fd5eb6782451c6f238851495eae7e47236e8936956d6c6f40907500dd"}
Oct 31 17:09:52.364984 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.364961    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:52.463652 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.463624    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca\") pod \"350406cf-db3a-491d-8a0d-79b0af35b6cd\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") "
Oct 31 17:09:52.463847 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.463724    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca\") pod \"350406cf-db3a-491d-8a0d-79b0af35b6cd\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") "
Oct 31 17:09:52.463847 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.463799    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb\" (UniqueName: \"kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb\") pod \"350406cf-db3a-491d-8a0d-79b0af35b6cd\" (UID: \"350406cf-db3a-491d-8a0d-79b0af35b6cd\") "
Oct 31 17:09:52.463988 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.463961    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca" pod "350406cf-db3a-491d-8a0d-79b0af35b6cd" (UID: "350406cf-db3a-491d-8a0d-79b0af35b6cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:09:52.464052 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.464041    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:52.465768 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.465744    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca" pod "350406cf-db3a-491d-8a0d-79b0af35b6cd" (UID: "350406cf-db3a-491d-8a0d-79b0af35b6cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:09:52.465894 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.465873    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb" pod "350406cf-db3a-491d-8a0d-79b0af35b6cd" (UID: "350406cf-db3a-491d-8a0d-79b0af35b6cd"). InnerVolumeSpecName "kube-api-access-nfljb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:09:52.565015 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.564936    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350406cf-db3a-491d-8a0d-79b0af35b6cd-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:52.565015 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:52.564967    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfljb\" (UniqueName: \"kubernetes.io/projected/350406cf-db3a-491d-8a0d-79b0af35b6cd-kube-api-access-nfljb\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:53.237529 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:53.237502    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb"
Oct 31 17:09:53.237714 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:53.237527    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-ngglb" event={"ID":"350406cf-db3a-491d-8a0d-79b0af35b6cd","Type":"ContainerDied","Data":"3193ab62df410be8bd8b3b858136f4e6560cbb5bd6ec7fe20ff4b31b536185e8"}
Oct 31 17:09:53.237714 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:09:53.237572    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3193ab62df410be8bd8b3b858136f4e6560cbb5bd6ec7fe20ff4b31b536185e8"
Oct 31 17:09:54.003700 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:09:54.003663    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:10:00.903523 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.903485    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"]
Oct 31 17:10:00.904021 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.903935    4519 cpu_manager.go:414] "containerMap: removing container" podUID="350406cf-db3a-491d-8a0d-79b0af35b6cd" containerName="revoke-keys"
Oct 31 17:10:00.904021 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.903950    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350406cf-db3a-491d-8a0d-79b0af35b6cd" containerName="revoke-keys"
Oct 31 17:10:00.904097 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.904059    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="350406cf-db3a-491d-8a0d-79b0af35b6cd" containerName="revoke-keys"
Oct 31 17:10:00.907539 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.907517    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:00.943323 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.943294    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265g7\" (UniqueName: \"kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:00.943517 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.943327    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:00.943517 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.943352    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:00.959805 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:00.959778    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"]
Oct 31 17:10:01.004524 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.004489    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:10:01.004673 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.004527    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:10:01.005281 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.005263    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:10:01.005326 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.005282    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tdnpj\""
Oct 31 17:10:01.044488 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.044439    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265g7\" (UniqueName: \"kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.044635 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.044492    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.044635 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.044518    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.045105 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.045079    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.046810 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.046790    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.098131 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.098108    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-265g7\" (UniqueName: \"kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7\") pod \"maas-api-revoke-sub-6b19338f-lnscd\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.221027 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.220994    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:01.585952 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:01.585877    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"]
Oct 31 17:10:01.592342 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:10:01.592311    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6535ba7_56b9_4cc9_a758_a31520cc8185.slice/crio-55bd65180e80b3ef65e7d0bad7ec6fa9c6f88733912cc84447a002aca0ec472f WatchSource:0}: Error finding container 55bd65180e80b3ef65e7d0bad7ec6fa9c6f88733912cc84447a002aca0ec472f: Status 404 returned error can't find the container with id 55bd65180e80b3ef65e7d0bad7ec6fa9c6f88733912cc84447a002aca0ec472f
Oct 31 17:10:02.296324 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:02.296168    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd" event={"ID":"f6535ba7-56b9-4cc9-a758-a31520cc8185","Type":"ContainerStarted","Data":"d19f3f7ca3ab250ba5c2f2f6d4307ecf78dea462611308ae126d55351805985f"}
Oct 31 17:10:02.296324 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:02.296245    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd" event={"ID":"f6535ba7-56b9-4cc9-a758-a31520cc8185","Type":"ContainerStarted","Data":"55bd65180e80b3ef65e7d0bad7ec6fa9c6f88733912cc84447a002aca0ec472f"}
Oct 31 17:10:02.405264 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:02.405210    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd" podStartSLOduration=2.4051976440000002 podStartE2EDuration="2.405197644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:10:02.403430173 +0000 UTC m=+4046.103359483" watchObservedRunningTime="2026-10-31 17:10:02.405197644 +0000 UTC m=+4046.105126954"
Oct 31 17:10:03.303709 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:03.303675    4519 generic.go:356] "Generic (PLEG): container finished" podID="f6535ba7-56b9-4cc9-a758-a31520cc8185" containerID="d19f3f7ca3ab250ba5c2f2f6d4307ecf78dea462611308ae126d55351805985f" exitCode=0
Oct 31 17:10:03.304108 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:03.303726    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd" event={"ID":"f6535ba7-56b9-4cc9-a758-a31520cc8185","Type":"ContainerDied","Data":"d19f3f7ca3ab250ba5c2f2f6d4307ecf78dea462611308ae126d55351805985f"}
Oct 31 17:10:04.441167 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.441142    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:04.479804 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.479772    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7\" (UniqueName: \"kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7\") pod \"f6535ba7-56b9-4cc9-a758-a31520cc8185\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") "
Oct 31 17:10:04.479975 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.479959    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca\") pod \"f6535ba7-56b9-4cc9-a758-a31520cc8185\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") "
Oct 31 17:10:04.480028 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.480017    4519 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca\") pod \"f6535ba7-56b9-4cc9-a758-a31520cc8185\" (UID: \"f6535ba7-56b9-4cc9-a758-a31520cc8185\") "
Oct 31 17:10:04.480340 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.480314    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca" pod "f6535ba7-56b9-4cc9-a758-a31520cc8185" (UID: "f6535ba7-56b9-4cc9-a758-a31520cc8185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:10:04.480478 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.480416    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-openshift-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:10:04.482314 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.482294    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca" pod "f6535ba7-56b9-4cc9-a758-a31520cc8185" (UID: "f6535ba7-56b9-4cc9-a758-a31520cc8185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:10:04.482398 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.482328    4519 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7" pod "f6535ba7-56b9-4cc9-a758-a31520cc8185" (UID: "f6535ba7-56b9-4cc9-a758-a31520cc8185"). InnerVolumeSpecName "kube-api-access-265g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:10:04.581432 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.581328    4519 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6535ba7-56b9-4cc9-a758-a31520cc8185-maas-api-xks-ca\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:10:04.581432 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:04.581367    4519 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265g7\" (UniqueName: \"kubernetes.io/projected/f6535ba7-56b9-4cc9-a758-a31520cc8185-kube-api-access-265g7\") on node \"ip-10-0-129-180.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:10:05.315135 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:05.315101    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd" event={"ID":"f6535ba7-56b9-4cc9-a758-a31520cc8185","Type":"ContainerDied","Data":"55bd65180e80b3ef65e7d0bad7ec6fa9c6f88733912cc84447a002aca0ec472f"}
Oct 31 17:10:05.315334 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:05.315160    4519 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bd65180e80b3ef65e7d0bad7ec6fa9c6f88733912cc84447a002aca0ec472f"
Oct 31 17:10:05.315334 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:05.315120    4519 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-lnscd"
Oct 31 17:10:13.699162 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:13.699129    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-jwxmj"]
Oct 31 17:10:23.863643 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:23.863609    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79c59c6ccb-v6mpc_eadb7dc3-b107-489b-88f2-f3f69567833b/manager/0.log"
Oct 31 17:10:24.003790 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:10:24.003755    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:10:24.473402 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:24.473374    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9w5mq_00b120bd-2b41-470e-bf2c-2a575adaba5c/manager/0.log"
Oct 31 17:10:27.674438 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:27.674406    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-ngglb_350406cf-db3a-491d-8a0d-79b0af35b6cd/revoke-keys/0.log"
Oct 31 17:10:27.889508 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:27.889481    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-lnscd_f6535ba7-56b9-4cc9-a758-a31520cc8185/revoke-keys/0.log"
Oct 31 17:10:28.096508 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:28.096476    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tklqt_43892db6-3779-41e1-a334-3093ca248a9b/revoke-keys/0.log"
Oct 31 17:10:30.513799 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:30.513767    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hs8rg_db6c0f50-71af-4052-925d-1dc7fb1d0327/manager/0.log"
Oct 31 17:10:31.535076 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:31.535043    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jwxmj_d3cf71a8-4583-48a6-990d-52e3afe1eeea/limitador/0.log"
Oct 31 17:10:33.089567 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:33.089540    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9d8tx_c269346d-fb60-416d-9f40-f3ef2530a5d5/discovery/0.log"
Oct 31 17:10:33.583097 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:33.583063    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-744f97f697-vjlc2_c5b308c3-b987-4fc1-a847-94a8fa7afac0/kube-auth-proxy/0.log"
Oct 31 17:10:33.816358 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:33.816333    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-5tkrq_f24f8e20-1030-4e69-96c6-a4a20a579947/istio-proxy/0.log"
Oct 31 17:10:34.268700 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:34.268667    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-d7kw8_76758c23-3740-4e77-9f9e-577afcda8bdf/istio-proxy/0.log"
Oct 31 17:10:35.263793 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:35.263764    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-r5grr_6cdf1158-9e56-4558-b7f5-2272af25e327/istio-proxy/0.log"
Oct 31 17:10:36.421653 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:36.421628    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7bknp_74e25e0f-6faa-453c-970f-72924dbd21bf/payload-pre-processing/0.log"
Oct 31 17:10:38.447601 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:38.447574    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2tnln_7f00984f-32ff-4b00-ae35-5372d3fd4ac3/main/0.log"
Oct 31 17:10:39.210932 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:39.210897    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-nxxwv_baf1235e-381e-406c-be7c-c19059168c8e/main/0.log"
Oct 31 17:10:39.482261 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:39.482181    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx_602afcaa-d57d-4bde-aadb-646f36b8733f/main/0.log"
Oct 31 17:10:39.711468 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:39.711412    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn_8ac757df-4216-40c2-8219-c01aade2b9f3/main/0.log"
Oct 31 17:10:49.834952 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:49.834859    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79c59c6ccb-v6mpc_eadb7dc3-b107-489b-88f2-f3f69567833b/manager/0.log"
Oct 31 17:10:50.445437 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:50.445407    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9w5mq_00b120bd-2b41-470e-bf2c-2a575adaba5c/manager/0.log"
Oct 31 17:10:53.050019 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:53.049990    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-ngglb_350406cf-db3a-491d-8a0d-79b0af35b6cd/revoke-keys/0.log"
Oct 31 17:10:53.277682 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:53.277654    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-lnscd_f6535ba7-56b9-4cc9-a758-a31520cc8185/revoke-keys/0.log"
Oct 31 17:10:53.497842 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:53.497814    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-tklqt_43892db6-3779-41e1-a334-3093ca248a9b/revoke-keys/0.log"
Oct 31 17:10:54.004179 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:10:54.004138    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:10:55.585643 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:55.585615    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hs8rg_db6c0f50-71af-4052-925d-1dc7fb1d0327/manager/0.log"
Oct 31 17:10:56.397396 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:56.397360    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jwxmj_d3cf71a8-4583-48a6-990d-52e3afe1eeea/limitador/0.log"
Oct 31 17:10:57.533709 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:57.533680    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9d8tx_c269346d-fb60-416d-9f40-f3ef2530a5d5/discovery/0.log"
Oct 31 17:10:57.988047 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:57.987995    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-744f97f697-vjlc2_c5b308c3-b987-4fc1-a847-94a8fa7afac0/kube-auth-proxy/0.log"
Oct 31 17:10:58.210396 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:58.210360    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-5tkrq_f24f8e20-1030-4e69-96c6-a4a20a579947/istio-proxy/0.log"
Oct 31 17:10:58.724635 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:58.724604    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-d7kw8_76758c23-3740-4e77-9f9e-577afcda8bdf/istio-proxy/0.log"
Oct 31 17:10:59.781258 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:10:59.781230    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-r5grr_6cdf1158-9e56-4558-b7f5-2272af25e327/istio-proxy/0.log"
Oct 31 17:11:01.031294 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:11:01.031261    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7bknp_74e25e0f-6faa-453c-970f-72924dbd21bf/payload-pre-processing/0.log"
Oct 31 17:11:02.552758 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:11:02.552730    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2tnln_7f00984f-32ff-4b00-ae35-5372d3fd4ac3/main/0.log"
Oct 31 17:11:03.271029 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:11:03.271003    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-nxxwv_baf1235e-381e-406c-be7c-c19059168c8e/main/0.log"
Oct 31 17:11:03.536793 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:11:03.536693    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7gkjx_602afcaa-d57d-4bde-aadb-646f36b8733f/main/0.log"
Oct 31 17:11:03.762912 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:11:03.762885    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ggpjn_8ac757df-4216-40c2-8219-c01aade2b9f3/main/0.log"
Oct 31 17:11:24.003871 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:11:24.003825    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:11:54.003889 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:11:54.003848    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:12:19.485017 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:19.484908    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5n9tx_28de26f0-048d-48c5-9df2-d1c4c33f4fb2/global-pull-secret-syncer/0.log"
Oct 31 17:12:22.253314 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:22.253278    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b4fr7_88955cf1-7c74-4198-a5a6-762fc37473f9/konnectivity-agent/0.log"
Oct 31 17:12:23.152783 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:23.152742    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-180.us-west-2.compute.internal_29bfc3859f40051644e6484149c81648/haproxy/0.log"
Oct 31 17:12:24.004682 ip-10-0-129-180 kubenswrapper[4519]: E1031 17:12:24.004647    4519 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jc94s" containerName="extractor"
Oct 31 17:12:29.252437 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.252400    4519 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"]
Oct 31 17:12:29.252927 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.252911    4519 cpu_manager.go:414] "containerMap: removing container" podUID="f6535ba7-56b9-4cc9-a758-a31520cc8185" containerName="revoke-keys"
Oct 31 17:12:29.252975 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.252929    4519 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6535ba7-56b9-4cc9-a758-a31520cc8185" containerName="revoke-keys"
Oct 31 17:12:29.253024 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.253012    4519 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6535ba7-56b9-4cc9-a758-a31520cc8185" containerName="revoke-keys"
Oct 31 17:12:29.256319 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.256300    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.306116 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.306088    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvx92\"/\"kube-root-ca.crt\""
Oct 31 17:12:29.306116 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.306102    4519 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvx92\"/\"openshift-service-ca.crt\""
Oct 31 17:12:29.349327 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.349293    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-podres\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.349520 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.349374    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cthxf\" (UniqueName: \"kubernetes.io/projected/5047f800-23d9-4842-bbac-305f0f2527ec-kube-api-access-cthxf\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.349520 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.349430    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-sys\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.349520 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.349497    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-proc\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.349647 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.349540    4519 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-lib-modules\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.353231 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.353208    4519 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kvx92\"/\"default-dockercfg-6qmmp\""
Oct 31 17:12:29.355501 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.355479    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"]
Oct 31 17:12:29.450402 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450371    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-podres\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.450611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450476    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthxf\" (UniqueName: \"kubernetes.io/projected/5047f800-23d9-4842-bbac-305f0f2527ec-kube-api-access-cthxf\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.450611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450513    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-sys\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.450611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450531    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-proc\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.450611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450564    4519 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-lib-modules\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.450611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450450    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-podres\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.450611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450589    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-sys\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.451027 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450630    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-proc\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.451027 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.450714    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5047f800-23d9-4842-bbac-305f0f2527ec-lib-modules\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.504826 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.504757    4519 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthxf\" (UniqueName: \"kubernetes.io/projected/5047f800-23d9-4842-bbac-305f0f2527ec-kube-api-access-cthxf\") pod \"perf-node-gather-daemonset-lmr46\" (UID: \"5047f800-23d9-4842-bbac-305f0f2527ec\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.569322 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.569287    4519 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:29.936126 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:29.936044    4519 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"]
Oct 31 17:12:29.942515 ip-10-0-129-180 kubenswrapper[4519]: W1031 17:12:29.942485    4519 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5047f800_23d9_4842_bbac_305f0f2527ec.slice/crio-2739dfbb0225e8fc46306732dbcb2661e2dc02a80628b8b9dd693aefd394a4e0 WatchSource:0}: Error finding container 2739dfbb0225e8fc46306732dbcb2661e2dc02a80628b8b9dd693aefd394a4e0: Status 404 returned error can't find the container with id 2739dfbb0225e8fc46306732dbcb2661e2dc02a80628b8b9dd693aefd394a4e0
Oct 31 17:12:30.032076 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:30.032045    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46" event={"ID":"5047f800-23d9-4842-bbac-305f0f2527ec","Type":"ContainerStarted","Data":"2739dfbb0225e8fc46306732dbcb2661e2dc02a80628b8b9dd693aefd394a4e0"}
Oct 31 17:12:30.190002 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:30.189921    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hs8rg_db6c0f50-71af-4052-925d-1dc7fb1d0327/manager/0.log"
Oct 31 17:12:31.039823 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:31.039779    4519 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46" event={"ID":"5047f800-23d9-4842-bbac-305f0f2527ec","Type":"ContainerStarted","Data":"4e3a6ae4cece93d1a0ae627e2d3ebc88823d894ba84f000f86fd865556a23afc"}
Oct 31 17:12:31.040304 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:31.040090    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:31.153611 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:31.153548    4519 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46" podStartSLOduration=2.153527896 podStartE2EDuration="2.153527896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:12:31.149315962 +0000 UTC m=+4194.849245275" watchObservedRunningTime="2026-10-31 17:12:31.153527896 +0000 UTC m=+4194.853457208"
Oct 31 17:12:31.511703 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:31.511671    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-jwxmj_d3cf71a8-4583-48a6-990d-52e3afe1eeea/limitador/0.log"
Oct 31 17:12:37.054805 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:37.054777    4519 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-lmr46"
Oct 31 17:12:39.249430 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:39.249383    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66558c4d44-cv8cg_aed3c8c6-be70-4af8-9349-e041cd22d75d/metrics-server/0.log"
Oct 31 17:12:39.668363 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:39.668290    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-s762m_efa8ea09-5a8d-4522-85eb-ae2922cedbb5/monitoring-plugin/0.log"
Oct 31 17:12:39.740072 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:39.740041    4519 scope.go:122] "RemoveContainer" containerID="bd1eb0df3efa8859325bdedbe7b6ec579e923c5e3103b5c1c4a18112b24f0b2c"
Oct 31 17:12:41.355472 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:41.355431    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kt6cf_605636d0-4624-47b5-b74a-8a4b09f582f5/node-exporter/0.log"
Oct 31 17:12:41.771587 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:41.771550    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kt6cf_605636d0-4624-47b5-b74a-8a4b09f582f5/kube-rbac-proxy/0.log"
Oct 31 17:12:42.188922 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:42.188842    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kt6cf_605636d0-4624-47b5-b74a-8a4b09f582f5/init-textfile/0.log"
Oct 31 17:12:43.859691 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:43.859662    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hkh4x_e69d240c-0010-405d-994a-6c819de2a2c7/kube-rbac-proxy-main/0.log"
Oct 31 17:12:44.276831 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:44.276803    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hkh4x_e69d240c-0010-405d-994a-6c819de2a2c7/kube-rbac-proxy-self/0.log"
Oct 31 17:12:44.694304 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:44.694174    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hkh4x_e69d240c-0010-405d-994a-6c819de2a2c7/openshift-state-metrics/0.log"
Oct 31 17:12:45.135785 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:45.135752    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/prometheus/0.log"
Oct 31 17:12:45.552310 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:45.552248    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/config-reloader/0.log"
Oct 31 17:12:45.969657 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:45.969635    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/thanos-sidecar/0.log"
Oct 31 17:12:46.389202 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:46.389092    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/kube-rbac-proxy-web/0.log"
Oct 31 17:12:46.807369 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:46.807332    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/kube-rbac-proxy/0.log"
Oct 31 17:12:47.225103 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:47.225076    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/kube-rbac-proxy-thanos/0.log"
Oct 31 17:12:47.646933 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:47.646863    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b09ab819-dfb8-46c1-a23e-53dcd927f337/init-config-reloader/0.log"
Oct 31 17:12:48.071790 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:48.071762    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-cpxsw_825b5af5-4574-43af-85d3-43ff9677289f/prometheus-operator/0.log"
Oct 31 17:12:48.485684 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:48.485656    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-cpxsw_825b5af5-4574-43af-85d3-43ff9677289f/kube-rbac-proxy/0.log"
Oct 31 17:12:49.326995 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:49.326967    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677b7f95bc-zl2nn_a18cb14a-22cf-4097-b27e-28cdb522de11/telemeter-client/0.log"
Oct 31 17:12:49.746119 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:49.746079    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677b7f95bc-zl2nn_a18cb14a-22cf-4097-b27e-28cdb522de11/reload/0.log"
Oct 31 17:12:50.164296 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:50.164193    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677b7f95bc-zl2nn_a18cb14a-22cf-4097-b27e-28cdb522de11/kube-rbac-proxy/0.log"
Oct 31 17:12:50.603094 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:50.603062    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867775f966-8rs4k_d65c6561-1883-49ed-9cd6-d29f927b4f0f/thanos-query/0.log"
Oct 31 17:12:51.019803 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:51.019776    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867775f966-8rs4k_d65c6561-1883-49ed-9cd6-d29f927b4f0f/kube-rbac-proxy-web/0.log"
Oct 31 17:12:51.437464 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:51.437380    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867775f966-8rs4k_d65c6561-1883-49ed-9cd6-d29f927b4f0f/kube-rbac-proxy/0.log"
Oct 31 17:12:51.853790 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:51.853762    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867775f966-8rs4k_d65c6561-1883-49ed-9cd6-d29f927b4f0f/prom-label-proxy/0.log"
Oct 31 17:12:52.370133 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:52.370100    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867775f966-8rs4k_d65c6561-1883-49ed-9cd6-d29f927b4f0f/kube-rbac-proxy-rules/0.log"
Oct 31 17:12:52.687343 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:52.687259    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9d8tx_c269346d-fb60-416d-9f40-f3ef2530a5d5/discovery/0.log"
Oct 31 17:12:52.792108 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:52.792078    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867775f966-8rs4k_d65c6561-1883-49ed-9cd6-d29f927b4f0f/kube-rbac-proxy-metrics/0.log"
Oct 31 17:12:53.006715 ip-10-0-129-180 kubenswrapper[4519]: I1031 17:12:53.006649    4519 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-744f97f697-vjlc2_c5b308c3-b987-4fc1-a847-94a8fa7afac0/kube-auth-proxy/0.log"